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Abstract. In the paper, properties of antisymmetric orbit functions are reviewed and 

■ further developed. Antisymmetric orbit functions on the Euclidean space E n are antisym- 
mctrized exponential functions. Antisymmctrization is fulfilled by a Weyl group, corre- 
sponding to a Coxeter-Dynkin diagram. Properties of such functions are described. These 
functions are closely related to irreducible characters of a compact semisimplc Lie group G 
of rank n. Up to a sign, values of antisymmetric orbit functions are repeated on copies 
of the fundamental domain F of the affine Weyl group (determined by the initial Weyl 

^ ■ group) in the entire Euclidean space E n . Antisymmetric orbit functions are solutions of 

\ the corresponding Laplace equation in E n , vanishing on the boundary of the fundamental 

■ domain F. Antisymmetric orbit functions determine a so-called antisymmetrized Fourier 
transform which is closely related to expansions of central functions in characters of irre- 

■ ducible representations of the group G. They also determine a transform on a finite set of 
| points of F (the discrete antisymmetric orbit function transform). Symmetric and antisym- 

■ metric multivariate exponential, sine and cosine discrete transforms are given. 
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^ ■ 1 Introduction 



In pQ and [2] we considered properties and applications of symmetric orbit functions (they are 
called there orbit functions; without the word "symmetric"). They are closely related to finite 
groups W of geometric symmetries generated by reflection transformations rj (that is, such 
that rf = 1), i = 1,2, ...,n, of the n-dimensional Euclidean space E n with respect to (n— 1)- 
dimensional subspaces containing the origin. In order to obtain a symmetric orbit function we 
take a point A S E n and act upon A by all elements of the group W . If O(A) is the Vl^-orbit of 
the point A, that is the set of all different points of the form wX, w £ W, then the symmetric 
orbit function, determined by A, coincides with 

4> x {x)= £ e2ni{fl,x) > 

where {fi, x) is the scalar product on E n . These functions are invariant with respect to the action 
by elements of the group W: 4>\{wx) = 4>\(x), w G W. If A is an integral point of E n , then 
<j)\(x) is invariant with respect to the affine Weyl group W aS corresponding to W. Since in the 
simplest case symmetric orbit functions coincide with the cosine function, sometimes they are 
called C-functions. 
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Symmetric orbit functions are multivariate functions having many beautiful and useful pro- 
perties and applicable both in mathematics and engineering. For this reason, they can be treated 
as special functions [2]. 

Symmetricity is the main property of symmetric orbit functions, considered in pQ and [2], 
which make them useful in applications. Being a modification of monomial symmetric functions, 
they are directly related to the theory of symmetric (Laurent) polynomials [3j [U [6] (see 
Section 11 in p]). 

Symmetric orbit functions 4>\{x) for integral A are closely related to the representation theory 
of compact groups G. In particular, they were effectively used for different calculations in 
representation theory [HJ [TUl EE] • They are constituents of traces (characters) of irreducible 
unitary representations of G. Although characters contain all (or almost all) information about 
the corresponding irreducible representations, they are seldom used as special functions. The 
reason is that a construction of characters is rather complicated, whereas orbit functions have 
much more simple structure. 

The symmetric orbit function 4>\(x) is a symmetrized (by means of the group W) exponential 
function e 27n ( A > x ') on E n . For each transformation group W, the symmetric orbit functions 
form a complete orthogonal basis in the space of symmetric (with respect to W) polynomials 
in e 2mX: > , j = 1, 2, . . . , n, or in the Hilbert space obtained by closing this space with respect to an 
appropriate scalar product. Orbit functions (fi\(x), when A runs over integral weights, determine 
so-called (symmetric) orbit function transform, which is a symmerization of the usual Fourier 
series expansion on E n . If A runs over the space E n , then <fi\(x) determines a symmetric orbit 
function transform, which is a symmetrization of the usual continuous Fourier expansion in E n 
(that is, of the Fourier integral). 

In the same way as the Fourier transform leads to discrete Fourier transform, the symmetric 
orbit function transform leads to a discrete analogue of this transform (which is a generalization 
of the discrete cosine transform [J_2]). This discrete transform is useful in many things related 
to discretization (see [I3j [TH HU CEl [J71 [J3J EE EU] ) . Construction of the discrete orbit function 
transform is fulfilled by means of the results of paper |21j . 

In this paper we consider antisymmetric orbit functions (since in the simplest case they 
coincide with the sine function, sometimes they are called S- functions). They are given by 

tp x ( x ) = (detw)e 2ni( ~ wX ' x \ x £ E n , 

where A is a strictly dominant weight and det w is a determinant of the transformation w (it is 
equal to 1 or —1, depending on either w is a product of even or odd number of reflections). The 
orbit functions (p\ have many properties that the symmetric orbit functions <j)\ do. But anti- 
symmetricity leads to some new properties which are useful for applications [22]. For integral A, 
antisymmetric orbit functions are closely related to characters of irreducible representations of 
the corresponding compact Lie group G. Namely, the character x\ °f the irreducible represen- 
tation T\ } A € P+, coincides with (p\ +p /(p p , where p is a half of a sum of positive roots related 
to the Weyl group W. 

Symmetric orbit functions are a generalization of the cosine function, whereas antisymmetric 
orbit functions are a generalization of the sine function. A generalization of the exponential 
functions are called .E-orbit functions. A detailed description of these functions for rank two see 
in [23]. 

Our goal in this paper is to bring together in full generality the diverse facts about antisym- 
metric orbit functions, many of which are not found in the literature, although they often are 
straightforward consequences of known facts. In general, for a given transformation group W 
of the Euclidean space E n , most of the properties of antisymmetric orbit functions, which are 
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described in this paper, are implications of properties either of the orbits of the group W or of 
the usual exponential function on E n . 

For dominant elements A, antisymmetric orbit functions y>\{x) are antisymmetric with re- 
spect to elements of the corresponding Weyl group, that is, ip\(wx) = (det w)<p\(x) for any 
w £ W. For this reason, antisymmetric orbit functions are defined only for strictly dominant 
elements A (a dominant element A is strictly dominant if wX = A means that w = 1). If A is 
integral strictly dominant element, then the corresponding antisymmetric orbit function <p\(x) 
is antisymmetric also with respect to elements of the affine Weyl group W , corresponding 
to the Weyl group W. Antisymmetricity is a main property of antisymmetric orbit functions. 
Because of antisymmetricity, it is enough to determine <p\(x) only on a fundamental domain of 
the affine Weyl group W aS (if A is integral). 

In the case when the group W is a direct product of its subgroups, say W = W\ x W2, the 
fundamental domain is the Cartesian product of fundamental domains for W\ and W%. Similarly, 
antisymmetric orbit functions of W are products of antisymmetric orbit functions of W\ and W2 ■ 
Hence it suffices to carry out our considerations for groups W which cannot be represented as 
a product of its subgroups (that is, for such W for which a corresponding Coxeter-Dynkin 
diagram is connected). 

In the article many examples of dimensions 2 and 3 are shown because they are likely to be 
used more often. 

We shall need a general information on Weyl groups, affine Weyl groups, root systems and 
their properties. We have given this information in pp. In order to make this paper self-contained 
we repeat a part of that information in Section [2j 

In Section [3] we describe signed Weyl group orbits. They differ from the Weyl group or- 
bits by a sign (equal to +1 or —1) assigned to each point of an orbit. To each signed orbit 
there corresponds an antisymmetric orbit function if a dominant element of the orbit is strictly 
dominant. 

Section U] is devoted to description of antisymmetric orbit functions. Antisymmetric orbit 
functions, corresponding to Coxeter-Dynkin diagrams, containing only two nodes, are given in 
an explicit form. In this section we also give explicit formulas for antisymmetric orbit functions, 
corresponding to Coxeter-Dynkin diagrams of A n , B n , C n and D n , in the corresponding ortho- 
gonal coordinate systems. In Section [5] properties of antisymmetric orbit functions are described. 
If A is integral, then a main property of the antisymmetric orbit function ip\ is an invariance 
with respect to the affine Weyl group W aS . We also give here the symmetric and antisymmetric 
orbit functions <j> p [x) and (p p (x), corresponding to the half-sum p of positive roots, in a form of 
products of the cosine and sine functions of certain angles depending on x. Specific properties 
of antisymmetric orbit functions of the Coxeter-Dynkin diagram A n are given in Section [6j 

In Section [7] we consider expansions of products of symmetric (antisymmetric) orbit functions 
into a sum of symmetric or antisymmetric orbit functions. These expansions are closely related 
to properties of (signed) VF-orbits. Many examples for expansions in the case of Coxeter-Dynkin 
diagrams A2 and C2 are considered. Section [8] is devoted to expansion of antisymmetric W-orbit 
functions into a sum of antisymmetric l^'-orbit functions, where W is a subgroup of the Weyl 
group W. Many particular cases are studied in detail. 

Connection between antisymmetric orbit functions fx(x) with integral A and characters of 
finite dimensional irreducible representations of the corresponding simple compact Lie groups 
is studied in Section [H In particular, the well-known Weyl formula for characters of such 
representations contains antisymmetric orbit functions. 

In Section [10] we expose antisymmetric orbit function transforms. There are two types of 
such transforms. The first one is an analogue of the expansion into Fourier series and the 
second one is an analogue of the Fourier integral transform. In Section [IT] a description of an 
antisymmetric generalization of the multi-dimensional finite Fourier transform is given. This 
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analogue is connected with grids on the corresponding fundamental domains for the affine Weyl 
groups W aS . Symmetric and antisymmetric multivariate exponential discrete transforms, as 
well as symmetric and antisymmetric multivariate sine and cosine discrete transforms are given 
in this section. 

In Section[12]we show that antisymmetric orbit functions are solutions of the Laplace equation 
on the corresponding ?i-dimensional simplex vanishing on a boundary of the simplex. It is shown 
that antisymmetric orbit functions are eigenfunctions of other differential operators. 

Section [13] is devoted to exposition of symmetric and antisymmetric functions, which are 
symmetric and antisymmetric analogues of special functions of mathematical physics or ortho- 
gonal polynomials. In particular, we find eigenfunctions of antisymmetric and symmetric orbit 
function transforms. These eigenfunctions are connected with classical Hermite polynomials. 



2 Weyl groups and affine Weyl groups 

2.1 Coxeter— Dynkin diagrams and Cartan matrices 

The sets of symmetric or antisymmetric orbit functions on the n-dimensional Euclidean space E n 
are determined by finite transformation groups W, generated by reflections rj, i = 1,2, ... ,n 
(a characteristic property of reflections is the equality rf = 1); the theory of such groups see, 
for example, in [23] and [25]. We are interested in those groups W which are Weyl groups 
of semisimple Lie groups (semisimple Lie algebras). It is well-known that such Weyl groups 
together with the corresponding systems of reflections rj, i = l,2,...,ra, are determined by 
Coxeter-Dynkin diagrams. There are 4 series and 5 separate simple Lie algebras, which uniquely 
determine their Weyl groups W . They are denoted as 

A n (n>l), B n (n>3), C n (n > 2), D n (n > 4), E 6 , E 7 , Eg, F 4 , G 2 . 

To these Lie algebras there correspond connected Coxeter-Dynkin diagrams. To semisimple 
Lie algebras (they are direct sums of simple Lie subalgebras) there correspond Coxeter-Dynkin 
diagrams, which consist of connected parts, corresponding to simple Lie subalgebras; these 
parts are not connected with each other (a description of the correspondence between simple 
Lie algebras and Coxeter-Dynkin diagrams see, for example, in |26j). Thus, we describe only 
Coxeter-Dynkin diagrams, corresponding to simple Lie algebras. They are of the form 



1 2 3 n " 

An 0-0-0----0 12 n " 3 ? n ~ l 

D n CH>--(MH) 




A diagram determines a certain non-orthogonal basis {a\,a 2 , ■ ■ ■ ,ce n } in the Euclidean spa- 
ce E n . Each node is associated with a basis vector a^, called a simple root. A direct link 
between two nodes indicates that the corresponding basis vectors are not orthogonal. Conversely, 
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an absence of a direct link between nodes implies orthogonality of the corresponding vectors. 
Single, double, and triple links indicate that the relative angles between the two simple roots 
are 2-7r/3, 37r/4, 5tt/6, respectively. There can be only two cases: all simple roots are of the same 
length or there are only two different lengths of simple roots. In the first case all simple roots 
are denoted by white nodes. In the case of two lengths, shorter roots are denoted by black nodes 
and longer ones by white nodes. Lengths of roots are determined uniquely up to a common 
constant. For the cases B n , C n , and F4, the squared longer root length is double the squared 
shorter root length. For G 2 , the squared longer root length is triple the squared shorter root 
length. 

If two nodes are connected by a single line, then the angle between the corresponding simple 
roots is 2tt/3. If nodes are connected by a double line, then the angle is 3tt/4. A triple line 
means that the angle is 5ir/6. Simple roots of the same length are orthogonal to each other or 
an angle between them is 2n/3. 

To each Coxeter-Dynkin diagram there corresponds a Cartan matrix M, consisting of the 
entries 



2{aj,a k ^ jjke{lj2 ,...,n}, (2.1) 



where {x, y) denotes the scalar product of x, y E E n . Cartan matrices of simple Lie algebras are 
given in many places (see, for example, [27]). We recall them here for ranks 2 and 3 because of 
their usage later on: 



Ao : \ „ x , Co: 



2 


-1 




-1 


2 


)■ 


2 


-1 





-1 


2 


-1 





-1 


2 





G 2 : 


2 -1 




-1 2 




-1 





2 -3 \ 
-1 2 J ' 

2-10 
-1 2 -1 
0-2 2 

Lengths of the basis vectors a, are fixed by the corresponding Coxeter-Dynkin diagram up 
to a constant. We adopt the standard choice in the Lie theory, namely 

(a, a) = 2 

for all simple roots of A n , D n , Eq, E7, E% and for the longer simple roots of B n , C n , F4, G 2 . 



2.2 Weyl group 

A Coxeter-Dynkin diagram determines uniquely the corresponding transformation group of E n , 
generated by reflections ri, i = 1,2, ... ,n. These reflections correspond to simple roots a,, 
i = 1,2, ... ,n. Namely, the transformation rj corresponds to the simple root oti and is the 
reflection with respect to (n — l)-dimensional linear subspace (hyperplane) of E n (containing 
the origin), orthogonal to a,;. It is well-known that such reflections are given by the formula 

TiX = x - 2 } X,ai ] ai, i = 1,2, . . . ,n, x E E n . (2.2) 

{ai,ai} 

Each reflection can be thought as attached to the i-ih node of the corresponding diagram. 

A finite group W, generated by the reflections r^, i = 1,2,... , n, is called a Weyl group, 
corresponding to a given Coxeter-Dynkin diagram. If a Weyl group W corresponds to a Coxeter- 
Dynkin diagram of a simple Lie algebra L, then this Weyl group is often denoted by W(L). 
Properties of Weyl groups are well known (see [21] and [25]). The orders (numbers of elements) 
of Weyl groups are given by the formulas 

\W(A n )\ = (n + 1)1, \W{B n )\ = \W(C n )\=2 n n\, \W{D n )\ = 2 n ~ l n\, 
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\W(E 6 )\ 
\W(F 4 )\ 



1152 



51840 



\W(E 7 )\ 
\W(G 2 )\ 



2 903 040, 



12. 



\W{E 8 )\ = 696 729 600, 



(2.3) 



In particular 



\W{A 2 )\ 



6 



\W(C 2 )\=8, 



\W{A 3 )\ = 24, 



\W(C 3 )\=4S. 



2.3 Roots and weights 

A Coxeter-Dynkin diagram determines a system of simple roots in the Euclidean space E n . 
Acting by elements of the Weyl group W upon simple roots we obtain a finite system of vectors, 
which is invariant with respect to W. A set of all these vectors is called a system of roots 
associated with a given Coxeter-Dynkin diagram. It is denoted by R. As we see, a system of 
roots R is calculated from simple roots by a straightforward algorithm. 

It is proved (see, for example, that roots of R are linear combinations of simple roots 
with integral coefficients. Moreover, there exist no roots, which are linear combinations of cti, 
i = 1,2, ... ,n, both with positive and negative coefficients. Therefore, the set of roots R can 
be represented as a union R = R + U where R + (respectively is the set of roots which 
are linear combination of simple roots with positive (negative) coefficients. The set i?+ (the 
set R-) is called a set of positive (negative) roots. 

As mentioned above, a set R of roots is invariant under the action of elements of the Weyl 
group W(R). However, wR + ^ R + if w is not a trivial element of W. The following proposition 
holds: 

Proposition 1. A reflection ri G W , corresponding to a simple root cii, maps cii into —ati and 
reflects the set R + \{ai} of all other roots of R+ onto itself. 

Let X a be the (n — l)-dimensional linear subspace (hyperplane) of E n which contains the 
origin and is orthogonal to the root a. Clearly, X a = X_ a . The set of reflections with respect 
to X a , a G coincides with the set of all reflections of the corresponding Weyl group W. 
The hyperplane X a consists of all points x G E n such that {x, a) = 0. 

The subspaces X a , a G R + , split the Euclidean space E n into connected parts which are 
called Weyl chambers. A number of Weyl chambers coincides with the number of elements of 
the Weyl group W. Elements of the Weyl group permute Weyl chambers. A part of a Weyl 
chamber, which belongs to some hyperplane X a is called a wall of this Weyl chamber. If for 
some element x of a Weyl chamber we have {x,a) =0 for some root a, then this point belongs 
to a wall. The Weyl chamber consisting of points x such that 

(x,cti) > 0, i = 1,2, ... ,n, 

is called the dominant Weyl chamber. It is denoted by D + . Elements of D + are called dominant. 
If (x, a.i) > 0, i = 1,2, ... ,n, then x is called strictly dominant element. 
The set Q of all linear combinations 




is called a root lattice corresponding to a given Coxeter-Dynkin diagram. Its subset 




is called a positive root lattice. 
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To each root a G R there corresponds the coroot a v defined by the formula 
2a 



a v 



(a, a) 

It is easy to see that a vv = a. The set Q v of all linear combinations 



gv = K- a . a v| a . GZ U0: 



l. i=l J i 

is called a coroot lattice corresponding to a given Coxeter-Dynkin diagram. The subset 



Q+ = {^2 a i a i I «i = 0, 1, 2, 



k i=i ) 

is called a positive coroot lattice. 

As noted above, the set of simple roots «j, i = 1, 2, . . . , n, form a basis of the space E n . In 
addition to the a-basis, it is convenient to introduce the so-called w-basis, ui,U2, ■ ■ ■ ,w n (also 
called the tesis of fundamental weights). The two bases are dual to each other in the following 
sense: 

2 -!p2^ = (a y ) = 5 • fc € {1,2,... (2.4) 

The a;-basis (as well as the a-basis) is not orthogonal. 

Note that the factor 2/{aj, a,-) can take only three values. Indeed, with the standard nor- 
malization of root lengths, we have 

o 

1 for roots of A n , D n , E 6 , E 7 , E$, 



2 

- = 1 for long roots of B n , C n , F4, G2, 

{oik, o^k) 
2 

= 2 for short roots of B n , C n , F±, 

{ak,a k } 

2 

= 3 for short roots of Gi- 

{ak,ak} 

For this reason, we get 

0% = afc for roots of A n , D n , Eq, E7, E$, 

a^ = a/j for long roots of B n , C n , F4, G2, 

0% = 2afc for short roots of B n , C n , F4, 

0% = 3ak for short roots of G-i- 

The a- and w-bases are related by the Cartan matrix (|2.ip and by its inverse: 



it 



Y,M jk uj k , iv j = Y,(M~ 1 )jka k - (2.5) 



k=l k=l 

For ranks 2 and 3 the inverse Cartan matrices are of the form 

M ' \ ( 1 2 ) ' C2 1 ( \ \ 2 J ' ° 2 : ( 1 2 
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Later on we need to calculate scalar products (x, y) when x and y are given by coordinates Xi 
and yi in o;-basis. It is given by the formula 

1 n 

i x > V) = 2 x 3Vk{ M ~ 1 )jk{oik I a k ) = xM~ x Dy T = xSy T , (2.6) 
j,k=i 

where D is the diagonal matrix diag (i(ai, at\), . . . , h(a n , a n )). Matrices S 1 , called 'quadratic 
form matrices', are shown in |27j for all connected Coxeter-Dynkin diagrams. 
The sets P and P+, defined as 

P = Zwi + • ■ ■ + Zw n D P+ = Z-° wi + • • • + Z-° L> n , 

are called respectively the weight lattice and the cone of dominant weights. The set P can be 
characterized set of all A G P n such that 



2(oy,A) _ 



(ay, ay) 



(a/, A) G Z 



for all simple roots ay. Clearly, Q C P. Below we shall need also the set P+ of dominant weights 
of P+, which do not belong to any Weyl chamber (the set of integral strictly dominant weights). 
Then A G Pf means that (A, a%) > for all simple roots «j. We have 



P+ = Z> V + Z >0 u} 2 + ■■■+ Z >0 oj n . 

The smallest dominant weights of P+, different from zero, coincide with the elements oji,U2, 
. . . ,u> n of the w-basis. They are called fundamental weights. They are highest weights of funda- 
mental irreducible representations of the corresponding simple Lie algebra L. 

Through the paper we often use the following notation for weights in u-basis: 

n 

z = ajujj = (ai ai ... a n ) , a± , . . . , a n G Z. 
If x = ^i a Ji then 

n 

(z,x) = ^a j 6 j . (2.7) 



2.4 Highest root 

There exists a unique highest (long) root £ and a unique highest short root £ s . The highest 
(long) root can be written as 

n n i . A n 

i=l i=l i=l 

The coefficients rrtj and can be viewed as attached to the i-th node of the diagram. They are 
called marks and comarks and are often listed in the literature (see, for example, [27] ). In root 
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systems with two lengths of roots, namely in B n , C n , Fa and G 2 , the highest (long) root £ is of 
the form 



B n 


c = 


(0 1 ... 0) = 


ai + 2a 2 + 2a 3 + • • • + 2a n , 


(2.9) 


C n 


e = 


(2 ... 0) = 


1a\ + 2a 2 H + 2a n -i + a n , 


(2.10) 


Fa 


c = 


(10 0) = 


2qi + 3a 2 + 4a3 + 2a4, 


(2.11) 


G 2 


e = 


(10) = 


2ai + 3a2- 


(2.12) 


n 

71, iy 


„, and £? n , all roots are of the same length, hence £ s = £. We have 




A n 




(10 ... 01) = 


Oil + a 2 H 1" «n, 


(2.13) 


D n 


£ = 


(010 ... 0) = 


ai + 2a 2 H h 2a n _ 2 + a n _i + a n , 


(2.14) 


E 6 




(0 1 ... 0) = 


ai + 2a 2 + 3«3 + 2a4 + Q5 + 2a 6 , 


(2.15) 


E 7 




(100 ... 0) = 


2ai + 3a 2 + 4«3 + 3«4 + 2as + ag + 20:7, 


(2.16) 


Es 




(0 ... 01) = 


2ai + 3a 2 + 4a3 + 5a4 + 6«5 + 4«6 + 2a7 + 3«s- 


(2.17) 



Note that for highest root £ we have 
Moreover, if all simple roots are of the same length, then 



(2.18) 



O' : 



ai. 



For this reason, 

(gi,<?2, • • • ,q n ) = (mi,m 2 , . . . ,m n ). 

for A n , D n and E n . Formulas (|2.13p - (|2.18p determine these numbers. For short roots Qj of B n , 
C n and Fa we have = 2a{. For short root a 2 of G 2 we have a 2 = 3a 2 . For this reason, 

(91, ?2, ••-,?n) = (1,2, .-.,2,1) for 
(9l,92,--.,9n) = (1,1,..., 1,1) for C n , 

(91,92,93,94) = (2,3,2,1) for F 4 , 
(91,92) = (2,1) for G 2 . 

To each root system i? there corresponds an extended root system (which is also called an 
affine root system). It is constructed with the help of the highest root £ of R. Namely, if 
a\, CK2, . . . , a n is a set of all simple roots, then the roots 

"0 := a>i,a 2 , ...,a n 

constitute a set of simple roots of the corresponding extended root system. Taking into account 
the orthogonality (non-orthogonality) of the root a® to other simple roots, the diagram of an 
extended root system can be constructed (which is an extension of the corresponding Coxeter- 
Dynkin diagram; see, for example, [28]). Note that for all simple Lie algebras (except for A n ) 
only one simple root is orthogonal to the root «o- I n the case of A n , the two simple roots a\ 
and a n are not orthogonal to oq. 
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2.5 Affine Weyl groups 

We are interested in antisymmetric orbit functions which are given on the Euclidean space E n . 
These functions are anti-invariant with respect to action by elements of a Weyl group W, which is 
a transformation group of E n . However, W does not describe all symmetries of orbit functions 
corresponding to weights A £ P+. A whole group of anti-invariances of antisymmetric orbit 
functions is isomorphic to the affine Weyl group which is an extension of the Weyl group W. 
This group is defined as follows. 

Let ai, «2j • • • j «n be simple roots in the Euclidean space E n and let W be the corresponding 
Weyl group. The group W is generated by reflections r ai , i = 1, 2, . . . , n. In order to construct 
the affine Weyl group W , corresponding to the group W, we have to add an additional 
reflection. This reflection is constructed as follows. 

We consider the reflection with respect to the (n — l)-dimensional subspace (hyperplane) 
A n _i containing the origin and orthogonal to the highest (long) root £, given in ()2.8|) : 

r € z = s-^^. (2.19) 

Clearly, rg £ W. We shift the hyperplane X n _\ by the vector £ v /2, where £ v = 2£/(£,£). 
(Note that by (|2.18p we have £ v = £. However, it is convenient to use here £ v .) The reflection 
with respect to the hyperplane A n _i + £ v /2 will be denoted by ro- Then in order to fulfill the 
transformation ro we have to fulfill the transformation r^ and then to shift the result by £ v , 
that is, 

r^x = rgx + £ v . 

We have r^O = £ v and it follows from (|2.19p that ro maps x + £ v /2 to 

r ( ( x + e v /2) + e v = x + e v /2 - ( x , e v >e 

Therefore, 

r (x + em = x + e/2 - = * + e v /2 - #l^ v 



x , * v /2 _ 2(x + e/2,e) y 2(^/2, Q 



Denoting x + £ v /2 by y we obtain that ro is given also by the formula 

roy = V + (l - e V = £ V + (2.20) 

The element ro does not belong to W since elements of W do not move the point £ i*^. 

The hyperplane A n _i + £ v /2 coincides with the set of points y such that r^y = y. It follows 
from (|2.20p that this hyperplane is given by the equation 

i= m =w= ^ fl * ft ' (2,2i) 

where 



k=l 



y = ^ a kUk, C = ^2qk 

k=l k=l 



(see (E2D). 
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A group of transformations of the Euclidean space E n generated by reflections ro, r ai , . . . , r an 
is called the affine Weyl group of the root system R and is denoted by W aS or by W^f (if is 
necessary to indicate the initial root system), see |28| . 

Adjoining the reflection ro to the Weyl group W completely change properties of the group 
W aS . 

If is the reflection with respect to the hyperplane X n _i, then due to (|2.19p and (|2.20j) for 
any x £ E n we have 

r r^x = r (r^x) = £ v + r^r^x = x + £ v . 

Clearly, (r^r^x = x + k£ v , k = 0, ±1,±2, that is, the set of elements (ror^) k , k = 
0, ±1,±2, ... , is an infinite commutative subgroup of W aS . This means that (unlike to the 
Weyl group W) is an infinite group. 

Since roO = ^ v , for any w G W we have 

wr = w£ v = 

where £^ is a coroot of the same length as the coroot £ v . For this reason, wr$ is the reflection 
with respect to the (n— l)-hyperplane perpendicular to the root £^ and containing the point £^/2. 
Moreover, 

(wr )r^x =x + ^w- 

We also have ((wro)r£v) k x = x + k^, k = 0, ±1, ±2, .... Since w is any element of W, then 
the set , w G W, coincides with the set of coroots of i? v , corresponding to all long roots of 
the root system R. Thus, the set • coincides with the lattice generated by coroots a v 
taken for all long roots a from R. 

It is checked for each type of root systems that each coroot ^ for a short root £ s of R is 
a linear combination of coroots u>£ v = £ w , w G W, with integral coefficients, that is, Q v = . 
Therefore, The set W aS • coincides with the coroot lattice Q v of R. 

Let Q y be the subgroup of W aS generated by the elements 

{wr )r w , w£W, (2.22) 

where r w = rgv for w G W. Since elements (|2.22p pairwise commute with each other (since they 
are shifts), Q v is a commutative group. The subgroup Q v can be identified with the coroot 
lattice Q v . Namely, if for g G Q v we have g ■ = 7 G Q v , then g is identified with 7. This 
correspondence is one-to-one. 

The subgroups W and Q v generate W aS since a subgroup of W , generated by W and Q v , 
contains the element ro. The group W aS is a semidirect product of its subgroups W and Q v , 
where Q v is an invariant subgroup (see Section 5.2 in [1] for details). 

2.6 Fundamental domain 

An open connected simply connected set D C E n is called a fundamental domain for the group 
^j/aff ^£ Qr g roU p ^ ft d oes not contains equivalent points (that is, points x and x' such 
that x = wx) and if its closure contains at least one point from each VF aff -orbit (from each 
W-orbit). It is evident that the dominant Weyl chamber (without walls of this chamber) is 
a fundamental domain for the Weyl group W. Recall that this domain consists of all points 
x = a\0J\ + • • • + a n uj n G E n for which 

a,i = (x, a^) > 0, i = 1, 2, . . . ,n. 
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We wish to describe a fundamental domain for the group W . Since W C W* tt , it can be 
chosen as a subset of the dominant Weyl chamber for W. 

We have seen that the element tq E W aS is a reflection with respect to the hyperplane 
X n -\ + £ v /2, orthogonal to the root £ and containing the point £ v /2. This hyperplane is given 
by the equation (|2.2ip . This equation shows that the hyperplane A" n _i + £ v /2 intersects the 
axices, determined by the vectors Ui, in the points uii/qi, i = 1,2, ... ,n, where qi are such as 
in (|2.2ip . We create the simplex with n + 1 vertices in the points 

0,^,...,^. (2.23) 

qi q n 

By the definition of this simplex and by (|2.2ip . this simplex consists of all points y of the 
dominant Weyl chamber for which (y,£) < 1. Clearly, the interior F of this simplex belongs to 
the dominant Weyl chamber. The following theorem is true (see, for example, [1]): 

Theorem 1. The set F is a fundamental domain for the afftne Weyl group W aS . 

For the rank 2 cases the fundamental domain is the interior of the simplex with the following 
vertices: 

A 2 : {0, wi, to 2 }, 
C 2 : {0, ui, uj 2 }, 
G 2 : {0, oj 2 }. 



3 Weyl group signed orbits 
3.1 Signed orbits 

As we have seen, the (n — l)-dimensional linear subspaces X a of E n , orthogonal to positive 
roots a and containing the origin, divide the space E n into connected parts, which are called Weyl 
chambers. A number of such chambers is equal to an order of the corresponding Weyl group W. 
Elements of the Weyl group permute these chambers. There exists a single chamber D+ such 
that (aj, x) > 0, x E -D+, i = 1, 2, . . . , n. It is the dominant Weyl chamber. 

Clearly, the cone of dominant weights P+ belongs to the dominant Weyl chamber D + . (Note 
that it is not a case for the set Q+.) We have P n D + = P + . 

Let y be an arbitrary dominant element of the Euclidean space E n , which does not lie on 
some Weyl chamber. We act upon y by all elements of the Weyl group W. As a result, we 
obtain a set of elements wy, w € W, which is called Weyl group orbit. All these elements are 
pairwise different. We attach to each point wy a sign coinciding with a sign of detw;. The set 
of all points wy, w E W, together with their signs is called a signed orbit of the point y with 
respect the Weyl group (or a Weyl group signed orbit, containing y). Points of signed orbits 
will be denoted by x + or x~, depending on a sign. Sometimes, we denote points wy, y E D + , of 
the signed orbit, containing the point y, as wy detw , where instead of a sign we have +1 or — 1, 
respectively. 

An orbit (where points do not have signs) of a point y E D + is denoted by 0(y) or 0\y(y)- 
A size of an orbit 0(y) is a number |0(y)| of its elements. Each Weyl chamber contains only 
one point of a fixed orbit Q(y). A signed orbit of a strictly dominant point y E E n is denoted 
by ± (y) or 0±{y). 

Note that orbits 0(y) are defined for any dominant elements y E E n . Signed orbits 0^(y) 
can be defined only for strictly dominant y E E n . 
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3.2 Signed orbits of A 1? A x x A l9 A 2 , C 2 , G 2 

Assuming that a > and 6 > 0, we list the contents of signed orbits in w-basis: 

A x : ± {a) B (a)+ (-a)- (3.1) 

Ai x : O i (a b) 3 (a 6)+, (-a b)~ , (a -6)", (-a -6)+ (3.2) 
A 2 : O i (a 6) B (a 6)+, (-a a+6)~, (a+b -by, 

(-6 -a)", (-a-6 a) + , (6 -a-6) + . (3.3) 

In the cases of C 2 and G 2 (where the second simple root is the longer one for C 2 and the shorter 
one for G 2 ) we have 

C 2 : ± (a 6) 9 (a 6)+, (-a a+6)~, (a+26 -b)~ , (a+2b -a-b) + , 

(-a -b) + , (-a -a-b)~, (-a-26 6)~, (-a-26 a+6)+, (3.4) 

G 2 : ± (a b) 3 ±(a &)+, ±(-a 3a+b)- , ±(a+6 -6)", 

± (2a+6 -3a-b) + , ±(-a-6 3a+2fe)+, ±(-2a-6 3a+2fe)~, (3.5) 

where db(c, d) + means two signed points (c, d) + and (— c, — d) + . 

As we see, for each point (c d) of a signed orbit of C 2 or G 2 there exists in the orbit the point 
(— c — d) with the same sign. 

3.3 The case of A n 

In the cases of Coxeter-Dynkin diagrams A n -\, B n , C n , D n , root and weight lattices, Weyl 
groups and signed orbits are described in a simple way by using the orthogonal coordinate 
system in E n . In particular, this coordinate system is useful under practical work with signed 
orbits. 

In the case A n it is convenient to describe root and weight lattices, Weyl group and antisym- 
metric orbit functions in the subspace of the Euclidean space E n+ i, given by the equation 

xi + x 2 H h x n+ i = 0, 

where x±, x 2 , . . . , a%+i are orthogonal coordinates of a point x E E n+ \. The unit vectors in 
directions of these coordinates are denoted by e^, respectively. Clearly, ej_Lej, i ^ j. The set of 
roots of A n is given by the vectors 

The roots 

ocij = ei - ej, i < j, 
are positive and the roots 

Oii = atij+i = e, - e i+ i, i = 1,2, . . . ,n, 
constitute the system of simple roots. 

n+1 

If x = ^2 Xiei, x\ + X2 + • • • + x n +i = 0, is a point of E n+ \, then this point belongs to the 

i=l 

dominant Weyl chamber Z)+ if and only if 
x\ > x 2 > ■ ■ ■ > x n +i- 
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Indeed, if this condition is fulfilled, then (x, cti) = Xi — Xi+\ > 0, % = 1,2, ... ,n, and x is 
dominant. Conversely, if x is dominant, then (x,cti) > and this condition is fulfilled. The 
point x is strictly dominant if and only if 

XI > X2 > ■ ■ ■ > X n +l- 

n 

If A = ^2 XiUJi, then the coordinates Aj in the w-coordinates are connected with the orthogonal 

8=1 

n+1 

coordinates mj of A = Yl m i e i by the formulas 

TL TL — 1 72 — 2 2 

m\ = — — Ai H — A 2 H — A 3 H 1 -~rK-i H tt A ™' 

n + 1 n + 1 n + 1 n + 1 n + 1 

1 n — In — 2 2 1 

m 2 = — Ai H — -A 2 H — rA 3 H 1 — -A n _i H — -A n , 

n+1 n + 1 n+1 n+1 n+1 

1 2 x n-2 x 2 x 1 

m 3 = — Ai — -A 2 H — -A 3 H 1 — -A n _i H — -X n , 

n+1 n + 1 n+1 n+1 n+1 

1 2 3 n-1 1 

m n = — Ai — A 2 —A3 - • • • —X n -i H — -A n , 

n + 1 n + 1 n + 1 n + 1 n + 1 

1 ^ 2 \ 3 x n ~ 1 \ n \ 
m n+1 — — Ai — A 2 —A 3 - • • • —An-! r~r A n- 

n + 1 n + 1 n + 1 n + 1 n + 1 

The inverse formulas are 

Xi = mi - m i+ i, i = l,2,...,n. (3.6) 

By means of the formula 

r a X = X —a (3.7) 

{a, a) 

for the reflection with respect to the hyperplane, orthogonal to a root a, we can find that the 

n+1 

reflection r ai . acts upon the vector A = ]P nijej, given by orthogonal coordinates, by permuting 

i=l 

the coordinates nij and rrij. Since for each i and j, l<i,j <n + l, there exists a root a^, t/ie 
VFey/ group W{A n ) consists of all permutations of the orthogonal coordinates mi, m 2 , . . . , m n +i 
0/ a poini A, i/iai is, VK(A n ) coincides with the symmetric group S n+ \. 

Sometimes (for example, if we wish that coordinates would be integers or non-negative inte- 
gers), it is convenient to introduce orthogonal coordinates X\, x 2 , . . . , x n+ \ for A n in such a way 
that 



xi + x 2 H 1- x n+ i = m, 



where m is some fixed real number. They are obtained from the previous orthogonal coordinates 
by adding the same number m/(n + 1) to each coordinate. Then, as one can see from (|3. 6j) . 
^-coordinates Aj = x% — ^Cj+i and the Weyl group W do not change. Sometimes, it is natural to 
use orthogonal coordinates x\, X2, ■ ■ ■ , x n+ \ for which all Xi are non-negative. 

We need below the half-sum p of the positive roots of A n , p = \ X] Q - ^ is easy to see that 

up to a common constant we have 
p = nei + (n - l)e 2 H he n , 
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that is, in orthogonal coordinates we have 

p = (n,n-l,...,l,0). (3.8) 
In the non-orthogonal w-coordinates we have 

p = LUi + LU2 H h UJ n - 

The signed orbit O i (A), A = (mi, rri2, ■ ■ ■ , m n +i), mi > rri2 > ••• > m n+ i, consists of all 
points 

(m m- m- Ygn(detw) 

obtained from (mi, m2, . . . , m n+ i) by permutations w £ W = S n+ i. Below instead of sgn (det w) 
we write simply det w. 

3.4 The case of B n 

Orthogonal coordinates of a point x £ E n are denoted by £1,22, • • • ,x n . We denote by ej the 
corresponding unit vectors. Then the set of roots of B n is given by the vectors 

a±i,±j = ±ej ± ej, i / j, a±j = ±e,, £ = 1, 2, . . . ,n 
(all combinations of signs must be taken). The roots 

<*i,±j = ei±ej, i<j, ati = ei, £ = l,2,...,n, 
are positive and n roots 

Oi{ . — e^ ej_j_i , i — 1,2,. ..,tx 1, ct n — e n 
constitute the system of simple roots. 

n 

It is easy to see that if A = ^ is a point of E n , then this point belongs to the dominant 

i=i 

Weyl chamber D+ if and only if 

mi > m.2 > • ■ ■ > m n > 0. 
Moreover, this point is strictly dominant if and only if 

mi > 7712 > • • • > m n > 0. 

n 

If A = ^i^ii then the coordinates A, in the cj-coordinates are connected with the coordi- 

i=l 

n 

nates mj of A = Yl m i e i by the formulas 

mi =Ai+A 2 H hA n _i+iA„, 

TO-2 = A2 + - • -+A n _i + |A n , 

m n = 2^ n ' 
The inverse formulas are 

Xi = mi-m i+ i, i = 1,2,... ,n- 1, A n = 2m n . 
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It is easy to see that if A G P+, then the coordinates mi, mi, ■ ■ ■ , m n are all integers or all 
half-integers. 

The half-sum p of positive roots of B n , p = \ Y a, in orthogonal coordinates has the form 

p = (n- in- §,..., i). (3.9) 

In ^-coordinates we have p = uj\ + uj2 + • • • + uj n ■ 

By means of the formula (|3.7p we find that the reflection r a acts upon orthogonal coordinates 

n 

of the vector A = Yl m i e i by permuting i-th and j-th coordinates if a = ±(ej — e,-), as the 

i=l 

permutation of i-th and j-th coordinates and the change of their signs if a = ±(e$ + e^), and 
as the change of a sign of i-th coordinate if a = ±e,. Thus, the Weyl group W{B n ) consists of 
all permutations of the orthogonal coordinates mi, mi, ■ ■ ■ , m n of a point A with possible sign 
alternations of any number of them. 

The signed orbit O i (A), A = (mi,m2, • • • , m n ), mi > m2 > • • • > m n > 0, consists of all 
points 

(±m h ,±m i2 ,...,±m ln ) detw (3.10) 

(each combination of signs is possible) obtained from (mi, mi, ■ ■ ■ , m n ) by permutations and 
alternations of signs which constitute an element w of the Weyl group W(B n ). Moreover, 
detw is equal to ±1 depending on whether w consists of even or odd number of reflections 
and alternations of signs. A sign of detw can be determined as follows. We represent w as 
a product w = es, where s is a permutation of (mi,m2, ■ ■ ■ ,m n ) and e is an alternation of 
signs of coordinates. Then det w = (det s)ei 1 €i 2 where det s is defined as in the previous 

subsection and is a sign of ij-th coordinate. 

3.5 The case of C n 

In the orthogonal system of coordinates of the Euclidean space E n the set of roots of C n is given 
by the vectors 

a±i,±j = ±e,±ej, i^j, a ±i = ±2e,,, i = l,2,...,n, 

where is the unit vector in the direction of i-th coordinate x% (all combinations of signs must 
be taken). The roots 

<*i,±j =ei± ej, i < j, on = 2ej, i = 1, 2, . . . , n, 
are positive and n roots 

a\ := ei - e i+ i, i = 1, 2, . . . ,n — 1, a n = 2e n 
constitute the system of simple roots. 

n 

It is easy to see that a point A = Yl miei G E n belongs to the dominant Weyl chamber D + 

i=l 

if and only if 

m l > ni2 > • • ■ > T71 n > 0. 

This point is strictly dominant if and only if 
mi > m2 > • • • > m n > 0. 
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If A = ^2 XiUi, then the coordinates Aj in the w-coordinates are connected with the coordi- 

i=l 

n 

nates mj of A = Yl m i e i by the formulas 

i=l 

mi =Ai+A 2 H hA n _i+A re , 

ni 2 = A 2 H hA„_i+A n , 

m n = A n . 

The inverse formulas are 

A, = mi - m i+ i, i = 1, 2, . . . , n - 1, A„ = m n . 

If A £ P+, then all coordinates mj are integers. 

The half-sum p of positive roots of C n , p = \ a i * n orthogonal coordinates has the form 

p = (n,n-l,...,2,l). (3.11) 



By means of the formula (|3.7f) we find that the reflection r a acts upon orthogonal coordinates 



n 



of the vector A = ^ ?Tiie, by permuting i-th and j-th coordinates if a = ±(ej — e^), as the 

permutation of i-th and j-th coordinates and the change of their signs if a = ±(ej + ej), and as 
the change of a sign of i-th coordinate if a = ±2ej. Thus, the Weyl group W(C n ) consists of all 
permutations of the orthogonal coordinates mi,m2, . . . ,m n of a point A with sign alternations 
of some of them, that is, this Weyl group acts on orthogonal coordinates exactly in the same 
way as the Weyl group W{B n ) does. 

The signed orbit O i (A), A = (mi, m,2, ■ ■ ■ , m n ), m\ > vt%2 > ••• > m n > 0, consists of all 
points 

(±m h , ±m i2 ±m in+1 ) dct w 

(each combination of signs is possible) obtained from (mi,ra2, . . . ,m n ) by permutations and 
alternations of signs which constitute an element w of the Weyl group W(C n ). Moreover, detw 
is equal to ±1 depending on whether w consists of even or odd numbers of reflections and 
alternations of signs. Since W(C n ) = W(B n ), then a sign of detw is determined as in the 
case B n . 

As we see, in the orthogonal coordinates signed orbits for C n coincide with signed orbits 
of B n . 

3.6 The case of D n 

In the orthogonal system of coordinates of the Euclidean space E n the set of roots of D n is given 
by the vectors 

a±vfcj = ±e,±ej, i / j, 

where ej is the unit vector in the direction of i-th coordinate (all combinations of signs must be 
taken). The roots 

oii,±j = ei±ej, i<j, 
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are positive and n roots 

OLi . — G{ Gj_|_i, i — 1, 2, ... , Tl 1, OLyi — G n —\ -\- G n 

constitute the system of simple roots. 

n 

It is easy to see that if A = ^ is a point of E n , then this point belongs to the dominant 

i=i 

Weyl chamber D + if and only if 

mi > ni2 > ■ ■ ■ > m n ~i > \m n \. 
This point is strictly dominant if and only if 

mi > 7712 > • ■ ■ > mn-1 > \ m n\ 
(in particular, m n can take the value 0). 

n 

If A = ^2 ^i^i, then the coordinates Aj in the w-coordinates are connected with the coordi- 

i=l 

n 

nates nij of A = Yl m i e i by the formulas 

mi =Ai+A 2 H hA n _ 2 +5(A n _i+A n ), 

m 2 = A 2 H hA n _2+§(A n _l+A re ), 



m n -i 



5(An-l + A n ), 



2 



— A n ), 



The inverse formulas are 

\i=mi-m i+ i, i = 1, 2, . . . ,re - 2, A n _i = m n _i + m n , A n = m n _i - m n . 

If A G P + , then the coordinates mi, m2, • • • , m n are all integers or all half-integers. 

The half-sum p of positive roots of D n , p = ^ ^ a, in orthogonal coordinates has the form 

p = (n-l,n-2, ...,1,0). (3.12) 
By means of the formula (|3.7p for the reflection r a we find that r a acts upon orthogonal 

n 

coordinates of the vector A = ^ mjej by permuting i-th and j-th coordinates if a = ±(ej — e^), 

i=l 

and as the permutation of i-th and j-th coordinates and the change of their signs if a = ± (ej + 
ej). Thus, the Weyl group W{D n ) consists of all permutations of the orthogonal coordinates 
mi,rri2, ■ ■ ■ , m n of a point A with sign alternations of even number of them. 

Since an alternation of signs of two coordinates Xi and Xj is a product of two reflections r a 
with a = (ei + e^) and with a = (ej — e^), a sign of the determinant of this alternation is plus. 
Note that \W(D n )\ = \\W{B n )\. 

The signed orbit ± (A), A = (mi,m2,...,m„), mi > m\2 > ••• > m n > 0, consists of all 
points 

{±?n h , ±m i2 ±m in+1 ) dct w 

obtained from (mi, m.2, • • • , m n ) by permutations and alternations of even number of signs which 
constitute an element w of the Weyl group W(D n ). Moreover, detw is equal to ±1 and a sign 
of detw is determined as follows. The element w G W(D n ) can be represented as a product 
w = ts, where s is a permutation from S n and r is an alternation of even number of coordinates. 
Then detw = dets. Indeed, a determinant of a transform, given by an element of W which is 
an alternation of two signs, is equal to +1 (since this element can be represented as a product 
of two reflections). 
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3.7 Signed orbits of A 3 

Signed orbits for A3, B3 and C3 can be calculated by using the orthogonal coordinates in 
the corresponding Euclidean space, described above, and the description of the Weyl groups 
W(A^), W(B^) and W(Cs) in the orthogonal coordinate systems. Below we give results of 
such calculations. Points A of signed orbits are given in the (^-coordinates as (a be), where 
A = auj\ + bu>2 + CUI3. 

The signed orbit ± (a b c), a > 0, b > 0, c > 0, of A3 contains the points 

± (a b c) B (a b c) + , (a+b -b b+c)~ , (a+b c -b-c) + , (a b+c -c)~, 

(a+b+c —c —b)~, (a+b+c —b—c b) + , (—a a+b c)~ , (—a 0+6+c — c) + , 

(6 —a—b a+b+c) + , (6+c —a— b—c a+b)~ , (—a—b a b+c) + , (—6 —a a+fe+c)~ 

and the points, contragredient to these points, where the contragredient of the point (a' 6' c') + 
is (— c' —6' — a') + and the contragredient of the point (a' 6' c')~ is (— c' —6' — a') - . 

3.8 Signed orbits of S 3 

As in the previous case, points A of signed orbits are given by the ^-coordinates (a be), where 
A = awi + buj2 + CLU3. The signed orbit ± (a b c), a > 0, b > 0, c > 0, of B3 contains the points 

± (a b c) 3 (ab c) + , (a+b -b 26+c)~, (-a a+b c)~, (6 -a-6 2a+2b+c) + , 

(-a-b a 2b+c) + , (-b -a 2a+2b+c)~ , (a b+c -c)~ , (a+b+c -b-c 2b+c) + , 
(—a a+b+c —c) + , (b+c —a— b—c 2a+2b+c)~ , (—a— b—c a 2b+c)~ , 
(-b-c -a 2a+2b+c) + , (-a-2b-c b c)~ ,(-a-b-c -b 2b+c) + , 
(a+2b+c -a-b-c c) + , (b a+b+c -2a-2b-c)~ , (a+b+c -a-2b-c 2b+c)~ , 
(-b a+2b+c -2a-2b-c) + , (-a-2b-c b+c -c) + , (-a-b -b-c 2b+c)~ , 
(a+2b+c -a-b -c)~ , (b+c a+b -2a-2b-c) + , (a+b -a-2b-c 2b+c) + , 
(-b-c a+2b+c -2a-2b-c)~ 

and also all these points taken with opposite signs of coordinates, signs of these points are also 
opposite. 

3.9 Signed orbits of C 3 

As in the previous cases, points A of signed orbits are given by the ^-coordinates (a be), where 
A = au)\ + bu)2 + cu>3 . The signed orbit O i (a b c), a > 0, b > 0, c > 0, of C3 contains the points 

± (a b c) B (ab c) + , (a+b -b b+c)~ , (-a a+b c)~ ,(b -a-b a+b+c) + , 

(-a-b a b+c) + , (-b -a a+b+c)~,(a b+2c -c)~, (a+b+2c -b-2c b+c) + , 
(-a a+b+2c -c) + , (b+2c -a-b-2c a+b+c)~ , (-a-b-2c a b+c)~ , 
(-b-2c -a a+b+c) + , (-a-2b-2c b c)~ ,(-a-b-2c -b b+c) + , 
(a+2b+2c -a-b-2c c) + , (b a+b+2c -a-b-c)~, (a+b+2c -a-2b-2c b+c)~ , 
(-b a+2b+2c -a-b-c) + , (-a-2b-2c b+2c -c)+, (-a-b -b-2c b+c)", 
(a+2b+2c -a-b -c)~ , (b+2c a+b -a-b-c) + , (a+b -a-2b-2c b+c) + , 
(-b-2c a+2b+2c -a-b-c)~ 

and also all these points taken with opposite signs of coordinates, signs of these points are also 
opposite. 
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4 Antisymmetric orbit functions 
4.1 Definition 

The exponential functions e 2m ( m < x ) ^ x G E n , with fixed m = (mi, ni2, ■ ■ ■ , m n ) determine the 
Fourier transform on E n . Antisymmetric orbit functions are an antisymmetrized (with respect 
to a Weyl group) version of exponential functions. Correspondingly, they determine an anti- 
symmetrized version of the Fourier transform. 

First we define symmetric orbit functions, studied in pQ. Let W be a Weyl group of transfor- 
mations of the Euclidean space E n . To each element A G E n from the dominant Weyl chamber 
(that is, (X,ai) > for all simple roots c^) there corresponds a symmetric orbit function <f>\ 
on E n , which is given by the formula 

cj> x {x) = Y, e2 " <At ' X> > x ^ E ^ (4- 1 ) 

where 0(A) is the VT-orbit of the element A. The number of summands is equal to the size 
|0(A)| of the orbit O(A) and we have A (O) = |0(A)|. 

Sometimes (see, for example, |14| and [E]), it is convenient to use a modified definition of 
orbit functions: 

$\(x) = \W x \Mx), (4.2) 

where W\ is a subgroup in W whose elements leave A fixed. Then for all orbit functions <f>\ we 
have <t> x (0) = \W\. 

Antisymmetric orbit functions are defined (see [22] and [29]) for dominant elements A, which 
do not belong to a wall of the dominant Weyl chamber (that is, for strictly dominant elements A). 
The antisymmetric orbit function, corresponding to such an element, is defined as 

<p x (x) = Y (detw)e 2ni ^ wX ' x \ x G E n . (4.3) 

w&W 

A number of summands in (|4.3h is equal to the size \W\ of the Weyl group W. We have 
<^ A (0) =0. 

Symmetric orbit functions <f)\ for which A G -P+ and antisymmetric orbit functions fx(x) for 
which A G -P+ are of special interest for representation theory. 

Example. Antisymmetric orbit functions for A±. In this case, there exists only one simple 
(positive) root a. We have (a, a) = 2. Then the relation 2(cj, a) /(a, a) = 1 means that 
(u>, a) = 1. This means that uj = a/2 and (to, to) = 1/2. Elements of coincide with mto, 
m G Z+. We identify points x of E\ = R with Ooj. Since the Weyl group W(Ai) consists of two 
elements 1 and r a , and 

r a x = x —a = x — tta = x — 2x = —x, 

(a, a) 

antisymmetric orbit functions (p\(x), A = muj, m > 0, are given by the formula 



Note that for the symmetric orbit function 4>\(x) we have <j)\ (x) = 2cos(7rm6>). 
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4.2 Antisymmetric orbit functions of A 2 

Antisymmetric orbit functions for the Coxeter-Dynkin diagrams of rank 2 were given in [22] . In 
this subsection we give these functions for Ai. 

Put A = aui + buj2 = (a b) with a > 0, b > 0. Then for ip\{x) = ip^ a b)(x) we have from (|4.3p 
that 

^ ^ = e 2iri{(a b),x) _ e 2iri{{-a a+b),x) _ e 2vri{(a+b -b),x) 

_|_ ^2iri((b —a—b),x) , ^2iri((—a—b a),x) ^2ni{(— b —a),x) 

Using the representation x = ipiai + 1P2&2, one obtains 

p b ^(x) = e 27T i{aipi+H2) _ e 2-wi(-atp 1 + {a+b)ip 2 ) _ e 27ri((a+b)t/>i -6V2) 

_|_ e 27ri(b?/)i-(a+fe)»/'2) _|_ e 2Tri((-a-b)4> 1 +aip 2 ) _ e 27ri(-iw/>i-ai/>2) ^4 4) 

The actual expression for ip^ a fy(x) depends on a choice of coordinate systems for A and x. 
Setting x = Q\uo\ + 62^2 and A as before, we get 

= e 3 i ((2a+6)0 1 + (a+26)0 2 ) _ e 2 f((-a+b)e 1 + (a+2b)e 2 ) 
_ e 2 f((2a+b)d 1 + (a~b)e 2 ) + e -2f((a-b)e 1 + {2a+b)e 2 ) 

+ g-^C^+s^ei+C-a+t)^) _ e -^§ L ((a+2b)e 1 +(2a+b)e 2 ) 
Note that (fr( a a )(x) are pure imaginary for all a > and 

f(a a)( x ) = 2i {sin 27ra(V>i +^2) + sin 2-^0(^1 — 2^ 2 ) — sin 27ra(2?/>i — ^2)} 

= 2i {sin 2vra((9i + (9 2 ) - sin 2ira0 1 - sin 27ra6> 2 } . (4.6) 

The pairs ip( a &)(x) + a )( x ) are always pure imaginary functions. 

4.3 Antisymmetric orbit functions of C2 and G 2 

Putting again A = au\ + 6w 2 = (06), x = 0\oji + #2<^2 and using the matrices S from (|2.6|) . 
which are of the form 

= 3 f j 2) ' 5 '( G '2) = I ^ 2) ' 
we find the orbit functions for C2 and G 2 : 

C 2 : y? (a 6) (a;) = 2 cos vr((a + 6)6»i + (a + 26)0 2 ) - 2 cos ir(Wi + (a + 26)6» 2 ) 

-2cosvr((a + 6)6'i + a6'2) + 2cos7r(66li - a6 2 ), (4.7) 

G 2 : 93 {a b) (x) = 2 cos vr((2a + 6)6>i + (a + §6)0 2 ) - 2 cos vr((a + b)0\ + (a + §6)6> 2 ) 
- 2 cos vr((2a + 6)6>i + (a + \b)6 2 ) + 2 cos vr((a + b)6i + ±60 2 ) 
+ 2cos7r(a6»i + {a+\b)6 2 ) - 2cos7r(a6»i - \W 2 ). (4.8) 



As we see, orbit functions for C 2 and G2 are real. 
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4.4 Antisymmetric orbit functions of A n 

It is difficult to write down an explicit form of orbit functions for A n , B n , C n and D n in 
coordinates with respect to the lj- or a-bases. For this reason, for these cases we use the 
orthogonal coordinate systems, described in Section [3j 

Let A = (mi,m,2, ■ ■ ■ , m n+ i) be a strictly dominant element for A n in orthogonal coordinates 
described in Subsection 13,31 Then m\ > mi > ••• > m n+ \. The Weyl group in this case 
coincides with the symmetric group S n +\. Then the signed orbit ± (X) consists of points 
(wX) detw , w £ W = S* n +i. Representing points x 6 E n+ i in the orthogonal coordinate system, 
x = (xi,X2, ■ ■ ■ , x n+ \), and using formula (|4.3p we find that 



<P\( X ) = 2^ (detw)e 



(Hpt „,v 27ri ^( mi '---' m ' i + i )'( :ri '---' ;:c ™+ 1 )) 

= (det y;) e 27ri (("' A )l 2 -'l+--- + ( 1 " A )n + ia:n + l) ) (4.9) 

WGSn+1 

where ((wA)i, (wX)2, ■ ■ ■ , (w\) n+ i) are the coordinates of the point wX. 

Note that the element — {m n+ i, m n , . . . , mi) is strictly dominant if the element (mi, ni2, ■ ■ ■ , 
m n+ i) is strictly dominant. In the Weyl group W{A n ) there exists an element wq such that 

w (m 1 ,m 2 , ■ ■ ■ ,m n +i) = (m n+1 ,m n , . . . ,mi). 
Moreover, we have 

det wo = 1 for A^-i and A^, 
det wo = —1 for ^Ufc+i and ^4fc+2- 

It follows from here that in the expressions for the orbit functions <P( mitm2l ,„ tmn+1 )(z) an d 
V-(m n+1 ,m„,-,m-f){ x ) there are summands 



e 



2-rri(w \,x) _ e 2iri{m n+1 x 1 -\ hmiXn+i) an( j e -2ir\{m n+1 xi-\ hmii„ + i) ^ 



respectively, which are complex conjugate to each other. Moreover, the first expression is con- 
tained with the sign (detu>o) in f( mi ,m2,...,m n+ i)( x )j that is, the expressions (|4.10p are contained 
in P(mi,m 2 ,.., m „ + i)(2:) and tp-( mn+1 ,m n ,..., mi ){ x ) with the same sign for n = 4k - 1,4k and with 
opposite signs for n = Ak + 1, 4k + 2, k £ Z + . 

Similarly, in the expressions (|4.9p for the function <P(mi,m2,...,m n +i)( x ) an d for the function 
V-K+i.mn,...^!)^) au other summands are (up to a sign, which depends on a value of n) 
pairwise complex conjugate. Therefore, 



¥ , (mi,m 2 ,... 1 m„ + i)( :r ) — ( / 5 -(m„ + i 1 m„,...,mi)( x ) (4-H) 

for n = 4k — 1,4k and 



( / 9 (mi,m 2 ,...,m n+ i)( :r ) — ^-(mn+i ,m„,...,mi) (4-12) 

for n = Ak + 1,4k + 2. 

If we use for A the coordinates Aj = {X,(x() in the w-basis instead of the orthogonal coordi- 
nates rrij, then these equations can be written as 



^(Ai ,...,A n ) ( x ) = V(A BJ ...,Ai)(aO, V(Ai ,...,A,j) (2-) = -<P(\ n ,...M)(x), 

respectively. According to (|4.11|) and (|4.12|) . if 

(mi,m 2 , ■ ■ ■ ,m n+1 ) = -(m n+1 ,m n , . . . ,mi) (4.13) 
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(that is, the element A has in the u;-basis the coordinates (Ax, A2, ■ ■ ■ , A2, Ai)), then the orbit 
function ip\ is real for n = 4k — 1,4k and pure imaginary for n = Ak + 1, Ak + 2. In the first 
case the antisymmetric orbit function can be represented as a sum of cosines of angles and in 
the second case as a sum of sines of angles multiplied by i = 



Proposition 2. In the orthogonal coordinates, antisymmetric orbit functions of A n can be 
represented as determinants of certain matrices: 



<P(mi,m 2 ,...,m n+1 )^) = det (e Mm4a *)«i 



/ g27rimi:Ei 



det 



3 27rim2Xi 



2mmiX2 



V e 



27rim n+ ixi e 27rim n+ iX2 



e 2mm 1 x n+1 \ 



(4.14) 



Proof. A proof of this formula follows from the fact that this expression for f(m 1 ,rn 2 ,...,rn n+1 )( x ) 
coincides with the expression given by the formula (|4.9|) : see |30| . ■ 



Taking into account the form of the half-sum of positive roots p for A n , we can write down the 
orbit function ip p (x), corresponding to the weight p = ^ Yl a, in the form of the Vandermonde 

a>0 

determinant, 



<p p {x) = det (e 2 ^)"^! = l[(e 2 ™» - e~ 2 ™<). (4.15) 

k<l 

The last equality follows from the expression for the Vandermonde determinant. 



4.5 Antisymmetric orbit functions of B n 

Let A = (mi, ttt-2, . . . , m n ) be a strictly dominant element for B n in orthogonal coordinates 
described in Subsection 13.41 Then m\ > rri2 > ■ ■ ■ > m n > 0. The Weyl group W(B n ) consists of 
permutations of the coordinates mj with sign alternations of some of them. Representing points 
x G E n also in the orthogonal coordinate system, x = {x\,X2, ■ ■ ■ ,x n ), and using formula (|4.3p 
we find that 

ip x {x)= J2 (detw)eie 2 ...e n e 2wi(u ' (£imi '"" £ " m " ) ' (a;i '-' x " )) 

£i = ±l w£Sn 

= S S ( det w ^ eiE2 ' ' ' e n e 2nii( - w( - eX))lXl+ - +(w{£X))nXn) , (4.16) 

Ei = ±l weS n 

where (w(e\))i, (w(eX)) n are the orthogonal coordinates of the points w(e\) if e\ — (simi, 

. . . , £n?17"ri) • 

Since in W(B n ) there exists an element which changes signs of all coordinates mi, then for 
each summand e 27n (( u, ( eA ))i :r iH h(w(e\)) n x n ) j n ^ e expressions (|4.16p for the antisymmetric orbit 
function ^( mii m 2v .. ]m „)(x) there exists exactly one summand complex conjugate to it, that is, the 
summand e -2m(((™(eA))izi+---+(u>(£A))„:z„)_ ip^jg summan(1 i s w ith sign (-1)™ = (detu/), where 
w 1 changes signs of all coordinates. Therefore, (det it/) = 1 if n = 2k and (detw') = —1 if 
n = 2k + 1. This means that antisymmetric orbit functions of B n are real if n = 2k and pure 
imaginary if n = 2k + 1. Each antisymmetric orbit function of B n can be represented as a sum 
of cosines of the corresponding angles if n = 2k and as a sum of sines, multiplied by i = 
if n = 2k + 1. The following proposition is true [30J: 
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Proposition 3. Antisymmetric orbit functions of B n can be represented in the form 

^ {mi ,m 2 ,...,m n )(x) = det (e 2 -™^ - e- w ^);. =1 = (2i)» det (sin 2 7 rm i x i )^ =1 . (4.17) 

Proof. Let us take on the right hand side of (|4.16|) the sum of terms with fixed w € S n . It can 
be written down as 

(det w) £ i £ 2 • • • £n e 2 ^ w{£X))lXl+ - +{w{eX))nXn) 

Si=±l 

— /^g^- w \ / g 27ri(wA)i xi _ g— 27ri(wA)ia;i\ , _ _ ^ e 27ri(«jA) n :c n _ e -27ri(uiA)„a; n ^ 

= (det w)(2i) n sin27r(u;A)ixi • . . . • sin 2ir{w\) n x n . 
Then for <p\(x) we have 

tp\(x) = (2i) n (det w) sin 27r(wA)iXi • . . . • sin 2it{wX) n x n 
weSn 

= (2i) n det (sm2irmiXj)™ j=1 , 
where A = (mi,m,2, ■ ■ ■ , m n ). Proposition is proved. ■ 

For the antisymmetric orbit function ip p , corresponding to the half-sum p of positive roots 
of B n , one has 



(p p {x) = (2i) n det (sin 2-TrpiXj) 



where p = (pi,p 2 , ■ ■ ■ , p n ) = (n - \, n - §, . . . , ~). 
4.6 Antisymmetric orbit functions of C n 

Let A = (mi,m,2, ■ ■ ■ ,m n ) be a strictly dominant element for C n in the orthogonal coordinates 
described in Subsection 13.51 Then m\ > ni2 > • • • > m n > 0. The Weyl group W{C n ) consists 
of permutations of the coordinates with sign alternations of some of them. Representing points 
x £ E n also in the orthogonal coordinate system, x = (x±, X2, ■ ■ ■ , x n ), we find that 



M*)= E E (detw)e 1 e 2 ...e n e 2 ^ 



(u)(eimi,...,e„m„),(ii,- ,x„)) 

£j=±l WGSn 



= E E ( det w ^ £l£2 ' ' ' e n e 2nii( - w( - eX))lXl+ - + ^ eX))nXn \ (4.18) 

£j=±l u>e5 n 

where, as above, (w(eX))i, . . . , (w(e\)) n are the orthogonal coordinates of the points w(eX) if 
eA = (eimi, . . . ,e n m n ). 

As in the case of B n , in the expressions (|4.18p for the functions ^( mii m 2 ,...,m„)( x ) f° r each 
summand e 2nl (( w ( £X ))^ x ^ h(w(e\))„x n ) fa ere exists exactly one summand complex conjugate to 
it, that is, the summand e - 2nl (( w ( £X ))i x i^ \-{w{e\)) n x n ) _ Moreover, this summand is with sign 
"+" if n = 2k and with sign "— " if n = 2k + 1. Therefore, antisymmetric orbit functions of C n 
are real if n = 2k and pure imaginary if n = 2k + 1. 

Note that in the orthogonal coordinates the antisymmetric orbit functions ^>( miy m 2 ,...,m n )( x ) 
of C n coincides with the antisymmetric orbit functions ^( mi , m2) ... ]mn )(x) of B n , that is, anti- 
symmetric orbit functions (|4.16p and (|4.18p coincide. However, a-coordinates of the element 
(mi,i7i2, ■ ■ ■ ,m n ) for C n do not coincide with a-coordinates of the element (mi,m2, . . . , m n ) 
for B n , that is, in a-coordinates the corresponding antisymmetric orbit functions of B n and C n 
are different. 
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Proposition 4. Antisymmetric orbit functions of C n can be represented in the form 

<P( mi ,m 2 ,..., mn )(x) = det {e 2 ™^ - e~ 2 ^ x % =1 = (2i)» det (sin2vrm i x i )^ =1 • (4.19) 

Proof. This proposition follows from Proposition [3] if we take into account that antisymmetric 
orbit functions of C n and of B n coincide in the orthogonal coordinate systems. ■ 

For the antisymmetric orbit function (p p , corresponding to the half-sum of positive roots 
of C n , one has 

ipp(x) = (2i) n det (sin27rpiXj)" iJ . =1 , 

where p = (pi, p 2 , . . . , p n ) = (n, n - 1, . . . , 1). 

4.7 Antisymmetric orbit functions of D n 

Let A = (mi,?7i2, . . . ,m n ) be a strictly dominant element for D n in the orthogonal coordinates 
described in Subsection 13.61 Then m\ > m 2 > ••• > m n _i > \m n \. The Weyl group W(D n ) 
consists of permutations of the coordinates with sign alternations of even number of them. 
Representing points x S E n also in the orthogonal coordinate system, x = (x±,x 2 , ■ ■ ■ ,x n ), and 
using formula (|4.3p we find that 



Ei = ±l weS n 

= J2' Yl (detw)e 2ni{{w{eX))lXl+ --- +iw{eX))nXn) , (4.20) 

£i = ±l weSn 

where (to(eA))i, . . . , (w(e\)) n are the orthogonal coordinates of the points w(e\) and the prime 
at the sum sign means that the summation is over values of £j with even number of sign minus. 
We have taken into account that an alternation of coordinates without any permutation does 
not change a determinant. 

Let m n 7^ 0. Then in the expressions (|4.20p for the orbit function <P( mitm2t ,.. jmn )i x ) °f ^n=2k 
for each summand e 2m i( w ( £X ))^ x ^ \-(w(e\))„x n ) there exists exactly one summand (with the same 
sign) complex conjugate to it. This means that these antisymmetric orbit functions of D 2 k are 
real. Each orbit function of D 2 k can be represented as a sum of cosines of the corresponding 
angles. 

It is also proved by using the formula (|4.20p that for m n ^ the antisymmetric orbit functions 
<P(m u ...,m 2k ,m 2k+1 )(x) and <P( mit ...,m 2 k -m 2 k+1 ){x) of D 2k +i are complex conjugate. 

If m n = 0, then it follows from (|4.20p that antisymmetric orbit functions of D n are real and 
can be represented as a sum of cosines of certain angles. 

Explicit forms of antisymmetric orbit functions of D n are described by the following propo- 
sition [30]: 



Proposition 5. Antisymmetric orbit functions of D n are representable in the form 

in, Jrr\— 1 rlpt ( f,2mmiXj _ -2mm l x j \ n ilA P t( p 2nim l Xj_ l _-2mm i Xj\ n 

^{m 1 ,m 2 ,...,m n )\ x ) ~ 2 aet \ e e Jij=l+2 aet l e +e )i,j=l 

= i(2i) n det(sin2vrm i x i )" i=1 + 2 n ^ 1 det(cos27rm J x j )™ i=1 (4.21) 
if m n 7^ 0, and in the form 

V( mi , m2 ,...,m n ){x) = \ det (e 2 ™^ + e~ 2 ^ m ^)l 3=l = 2 n ~ l det (cos 2nm i x j )^ =l (4.22) 
ifm n = 0. 
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Proof. Let m n ^ 0. We take on the right hand side of (|4,20p a sum of terms with fixed w £ S n . 
It can be written as 

I w = (detw) ' £l e 2 ■ ■ ■ e n e 2ni « w ( eX » lXl+ --- +( - w( - eX » nXn \ 

e<=±l 

where £i£2 = 1 (since there is an even number of £j with £, = — 1). A value of I w does not 
change if we add and subtract the same term to it: 



I w = (detw) ' £l e 2 ■ ■ ■ e n e 2ni{ ^ eX))lXl+ --- +( - w(£X))nXn) 

Si=±l 

+ |(detw) Y " eie 2 ■ ■ ■ e n e 27Ti ^ w( - £X))lXl+ - +{w{£X))nXn) 

Si=±l 

- l(detw) Y % l£2 ■ ■ ■ e n e 2ni ^ wieX))lXl+ --- +[w{eX))nXn) , 



Ei=±l 

the sum with two primes here means that the summation is over values of £j with an odd number 
of negative £j. We split down the right hand side into two parts, 

l(detw)^2' ^(detw) ■■■ + |(detw) ^ ' h |(detw) ^ ' • • • 

£j=±l £;=±1 J L £i=±l Ej=±l 

and repeat the reasoning of the proof of Proposition [3j As a result, we have 

-(det w)(e 2 ' K ^ wX ^ lXl — g -27 ™^)!^!) . . . ^g27ri(«)A) n a: n _ e -2m{w\) n x n -j 



27ri(tuA)ia;i \ , , _ ( 2m(w\) n Xr, 



•■w — 2' 

' ^ * * * 

^ (det w)(2i) n sin27r(u;A)i:Ei sin 27r(wA) n a; ri 

+ i(det w;)2 n cos 27r(u;A)iXi cos 2it(w\) n x n . 



+ e 



-2ni(w\) 



) 



Then for ip\(x) we have 

ip\(x) = ^(2i) n ^2 (det sin 27r(-u;A)iXi sin27r(wA) n x n 

+ \l n ^2 (det w) cos 27r(-u;A)iXi cos 2ir(w\) n x n 



i(2i) n det (sin27rm i a; j )" . =1 + 2 n_i det (cos 2TTm i x j )^ j 



where A = (mi, 7712, . . . , 77t n ). Thus, the proposition is proved for the case m n 7^ 0. 

Let m n = 0. Then, in the expression (|4.20p for (p\(x), in each term of the sum there exists 
the multiplier e 211 ^" 1 " 1 * with some i. Since m n = we have e 2 ™"" 1 ™ 1 * = 1 for e n = 1 and for 
e n = —1. The case e n = 1 gives an even number of negative £j in the set (£1, £2, • • • , £ n -i) an d 
the — 1 gives an odd number of negative £, in this set. Therefore, we may throw out 

the sum over e n and consider that summation in (|4.20p runs over e% = dbl, i = 1,2, ... n — 1 
(with even or odd number of negative £j): 



tp x (x)= Y Y(detw)e 27ri ^ w( - £X))lX1+ --- +{w{eX))nX " ) , 

Ei=±l wes n 



(4.23) 



where in the sum the multipliers, containing m n , are removed and the first sum does not contain 
summation over s n . As in the case of the proof of Proposition [3| we take in (|4.23j) terms with 
fixed w and write down it as 



(detw;) ^ ^ 



((w(e\))ixi-\ \-(w(e\)) n x„) 
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= 2 n ~ 1 (det w) cos 2ir(w\)iXi cos 27r(w\) n x n , 

where, as before, the multipliers, containing m n , are omitted. Since e 2mm ^ x i _|_ e -2mmnXi _ 2^ 
this leads to the formula (|4.22p for <p\(x) when m n = 0. The proposition is proved. ■ 

For the antisymmetric orbit function ^, /) = ^ a, one has 

Q>0 

(p p (x) = 2 n ~ 1 det (cos27r piXj)™ j=1 , 
where p = (pi,p 2 , • • • , Pn) = (n - 1, n - 2, . . . , 1, 0). 

4.8 Symmetric orbit functions of B n , C n and D n 

In Subsections I4.5H4.7I we have derived expressions for antisymmetric orbit functions (p\(x) 
of B n , C n and D n in orthogonal coordinates as determinants of sine and cosine functions. 
Similar expressions can be derived for symmetric orbit functions of B n , C n and D n . Since in 
the defining expressions for symmetric orbit functions <p\(x) (see formula (|4.ip ) there are no 
multipliers (detw), then 4>\{x) is expressed in terms of products of sine and cosine functions 
(instead of determinants). 

As before, we express elements A = (1711,1712, ■ ■ ■ , m n ) and x = (x\,X2, ■ ■ ■ , x n ) in the corre- 
sponding orthogonal coordinate systems. In Propositions [6] and [7] below we suppose that A is 
an element of the dominant Weyl chamber and is not obligatory integral. 

Proposition 6. Symmetric orbit functions of B n and C n can be represented in the form 

0(m 1; m 2 ,...,m„)(^) = 2 ™ cos27Tm w(i)Xi cos 2vrm u ,( n , ) x n , (4.24) 

WGSn 

where w(l),w(2), . . . , w(n) is the set of numbers 1,2, ... ,n obtained by acting by the permutation 
w G S n . 

This proposition is proved in the same way as Proposition [3] and we omit it. 
Proposition 7. Symmetric orbit functions of D n can be represented in the form 

4>( mi , m2 ,...,m n )(x) = 2 n_1 ^2 cos2vrm w(1) a;i cos 2nm w{n) x n 

+ ^(2i) n sin27rm„,( 1 )Xi sin 2 r Km w ^x n (4-25) 

WGSn 

if m n 7^ and in the form 

r(mi ,1712, ■■■,m n 

)(x)=2 n 1 cos27r?n u ,( 1 )Xi cos 2irm w ^ n ^x n (4.26) 

WGSn 

if fUn — 0, where w(l),w(2), . . . , w(n) is the set of numbers 1,2, ... ,n obtained by acting by the 
permutation s £ S n . 

A proof is similar to the proof of Proposition 
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5 Properties of antisymmetric orbit functions 

5.1 Anti-invariance with respect to the Weyl group 

Since the scalar product (•, •) in E n is invariant with respect to the Weyl group W, that is, 

(wx, wy) = (x, y), weW, x,y G E n , 

antisymmetric orbit functions ip\ for strictly positive elements A (not obligatory belonging to P^) 
are anti-invariant with respect to W, that is, 

(f\{w'x) = (det w')ip\(x), w' € W. 

Indeed, 

tp x (w'x) = {detw)e 2ni{wX ' w ' x) = (detw)e 2ni{w '~ lwX > x) 
wew weW 

= (detw') {detw)e 27Ti{wX ' x) = (detw')<p x (x) 

since w'~ 1 w runs over the whole group W when w runs over W . 

5.2 Anti-invariance with respect to the affine Weyl group 

The affine Weyl group W is generated by reflections ro, r ai , . . . , r an (see Subsection [23]) . We 
say that an antisymmetric orbit function ip x , A € P+ , is anti-invariant with respect to the affine 
Weyl group W aS if ip\(rox) = —ip x (x). Let us show that <p\(x) satisfies this relation. Since 
tqx = r^x + £ v , where £ is the highest root (see Subsection I2.4p . for fi £ P we have 

(//, r x) = (//, £ v + r^x) = 2< f'/p + (/i, r^x) = integer + (r^fi, x) 
since r| = 1. Hence, 

V X (r x) = ^2 (detw)e 2wl{wX ' rox) = ^ (detw)e 27Ti{r ^ X ' x) 



(detrj) (detw)e 2ni{wX > x) = -<px(x) 



wew 

since r^ is a reflection belonging to W. 

If A P, then i/?^ is not anti-invariant with respect to ro. It is anti- invariant only under 
action by elements of the Weyl group W. 

Due to the anti-invariance of antisymmetric orbit functions ip\, A £ Pf, with respect to the 
group W aS , it is enough to consider them only on the fundamental domain F = FiW^) of VF aff . 
Values of ip\ on other points of E n are determined by using the action of W on F or taking 
a limit. In particular, functions ip\, A € PJjT, are anti-invariant under a reflection with respect 
to any (n — 1) -dimensional wall of the fundamental domain F. 



5.3 Continuity and vanishing 

An antisymmetric orbit function <p x is a finite sum of exponential functions. Therefore it is 
continuous and has continuous derivatives of all orders in E n . 

Due to anti-invariance of the orbit functions (p\, A € D\ , with respect to the Weyl group W, 
(fx vanishes on all walls of the Weyl chambers. The anti-invariance (p x (rQx) = —ip x {x) for 
A € P^ shows that if\, A £ Pj^, vanishes on the boundary of the fundamental domain F of the 
affine Weyl group W aS . 
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5.4 Realness and complex conjugation 

The results, formulated below and concerning antisymmetric orbit functions of the Coxeter- 
Dynkin diagrams A n , B n , C n , D n and G2, were proved in the previous section. 

Antisymmetric orbit functions of the following Coxeter-Dynkin diagrams are real: 

-E>2fc, C2k, G<i. 

Antisymmetric orbit functions of the Coxeter-Dynkin diagrams E>2k+\ and C2k+i are purely 
imaginary. The antisymmetric orbit functions ip\ of the Coxeter-Dynkin diagrams A^k—i and 
A^k satisfy the condition 



P(Al ) Aa,... I A«)0»0 = <P(\n,\n-l,.,\l)( X ) 

and antisymmetric orbit functions ip\ of the Coxeter-Dynkin diagrams A^k+i and Aik+2 satisfy 
the condition 



¥>(Ai,A 2 ,..,A n )0*0 = — y(A n ,A n _i,...,Ai) i x ) ■ 

Antisymmetric orbit functions ^(a 1 ,a 2 ,...,a„) of D n are real if X n -i = X n . If X n -i / A n , then 
orbit functions v 3 (Ai,A 2 ,...,A n ) are real for n = 2k and satisfy the condition 

( ^(Ai,...,A n _2,A n _i,A n ) ¥ , (Ai,...,A n _2,An,A 7l _i) 

if n = 2k + 1. 

5.5 Scaling symmetry 

Let O(A) be an orbit of A, A G D~\_. Since w(cX) = cw(X) for any c G R and for any w G W, 
then the orbit O(cA) is an orbit consisting of the points cwX, w G W. Moreover, points wX 
and cwX of the signed orbits ± (A) and ± (cA), respectively, have the same sign. Let tp\ = 
(det w)e 2mwX be the antisymmetric orbit function for A G D~f. Then 

w€W 

VcX ( x ) = ^ {detw)e 2 ^ cwX ^ = (detw)e 2 ^ wX ' cx ^ = <p X {cx). 

The equality ip c \(x) = ip\(cx) expresses the scaling symmetry of orbit functions. 

If we consider only antisymmetric orbit functions tp\, corresponding to A G P^ , then the 
scaling symmetry ip c \(x) = ip\(cx) holds for those values c G R\{0} for which cA G Pt. 

5.6 Duality 

Due to the invariance of the scalar product (•, •) with respect to the Weyl group W, {wfj,, wy) = 
{jjL, y), for x G E n not lying on a wall of some Weyl chamber we have 

tp x (x) = ^2 (detw)e 2ni ^ w ' lx ^ = ^2 {detw)e 27vi( ~ X ' wx ^ = tp x (X). 
The relation <p\{x) = tp x {X) expresses the duality of antisymmetric orbit functions. 
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5.7 Orthogonality 

Antisymmetric orbit functions ^, A £ P~t, are orthogonal on F with respect to the Euclidean 
measure I3T1: 



- 1 J <p x (x)<px(x)dx = \W\5 xx >, (5.1) 



where the overbar means complex conjugation. This relation directly follows from orthogonality 
of the exponential functions e 2nl ^' x ^ (entering into the definition of orbit functions) for different 
weights \i and from the fact that a given weight v G P belongs to precisely one orbit function. 
In (|5.ip . \F\ means an area of the fundamental domain F. 

Sometimes, it is difficult to find the area \F\. In this case it is useful the following form of 
the formula (|5,1D : 



T 



(px(x)cpx>(x)dx = \W\5xy, 



where T is the torus in E n described in Subsection 19.11 below. If to assume that an area of T is 
equal to 1, |T| = 1, then \F\ = \W\ _1 and formula (|5.ip takes the form 

J (px(x)(px'(x)dx = 8x\>. (5.2) 

The formula (|5.2p gives the orthogonality relation for the antisymmetric multivariate sine 
function (|4.19[) . We have 

2 2n / det (sin 2imiiXj)™ . =1 det (sin 2irm' i Xj')™ dx = 5 m ,m') (5-3) 
Jf m— 

where m = (mi, ni2, ■ ■ ■ , m n ) and m' = (m' 1 ,m' 2 ,...,m'J are strictly dominant and integral 
(that is, nii,m'j £ Z >0 , mi > ni2 > ••• > m n > 0), and the domain F consists of points 
x = (x±,X2, • • • , x n ) G E n such that 

— > X\ > X2 > • • • > x n > 

(see Subsection 15.101 below) . 

A similar orthogonality relation can be written down for the symmetric multivariate cosine 
function (|4.24p . Introducing the notation 

det + (cos 2irmiXj)™j =1 := cos 2-Km w ^xi cos 2irm w ^ n )X n (5-4) 

we have 

2 2n / det + (cos 2irrriiXj)™ ._ 1 det + (cos 2irm' i Xj) n . =1 dx = \ W m \5 m in ', (5-5) 
Jf 

where m and m' are such that mi > rri2 > • • • > m n , m! 1 > m' 2 > • • • > m' n , and \W m \ is an 
order of the subgroup |W m | C S n consisting of elements leaving m invariant. 
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5.8 Orthogonality to symmetric orbit functions 

Let on be a simple root. We create the domain F = F U r ai F, where r ai is the reflection 
corresponding to the root an and F is the fundamental domain. Since for p £ P+ we have 

where <^„ is a symmetric orbit function, then 



(j) ll (x)(p fJi (x)dx = 0. (5.6) 

^GXt 

Indeed, due to symmetry and antisymmetry of symmetric and antisymmetric orbit functions we 
have 



4> fl (x)if ll (x)dx = / (f) tl (x)ip fl (x)dx + / 4> fl (x)(f /1 (x)dx 

F cxt JF Jr a .F 



4 , ^(x)(fi^(x)dx + / 4> fl (x){-(p tl {x))dx = 0. 

For the case of A\ the orthogonality (15. 6p means the orthogonality of the functions sine and 
cosine on the interval (0, 2ir). 

The formula (15. 6p determines orthogonality of multivariate sine and cosine functions (I4.19h 
and (15. 



/ det (sin 2irmiXj)™ - =1 det + (cos 27rmjXj)™, =1 <ix = 0, 

where the notations are such as in (|5.3p and P cxt consists of points cc G E n such that 
3 > x\ > X2 > ■ ■ ■ > x n > or ^ > X2 > X\ > Xs > X4 > ■ ■ ■ > x n > 0. 

5.9 Antisymmetric orbit functions cp p 

Let p be the half-sum of all positive roots of a root system: 

It is well-known that for all simple Lie algebras in w-coordinate we have 

p = U3\ + UJ2 H hW„= (1 1 • • • 1). 

The antisymmetric orbit function (p p is important in the theory of characters of group represen- 
tations. We have 

(p p (x) = J2( detw ) e2 ™ {wp ' x) - ( 5 - 8 ) 

Proposition 8. The antisymmetric orbit function ip p can be represented as 
(p p (x) = Y[{e ni{a ' x) - e -™ {a ' x) ) = e 2ni{p ' x) Y[{1- e- 2wi{a ' x) ) 

= (2i) r Yl sinvr(a,x), (5.9) 
where r is a number of positive roots in the corresponding root system. 
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Proof. The expression g(x) = \\ (e 7 ™^'^ — e~ m ( a ' x ') is antisymmetric with respect to the 

Weyl group W. Indeed, if is a reflection, corresponding to the simple root ccj, then due to 
Proposition [1] we have 



g( n x) = JT( e ' ri < a ' r * a, > - e -wi(«.»-i*>) = JJ^faa.*) _ e - 



-7ri<rja,a;) \ 

a>0 a>0 
— *ri(ai,x) _ e 7ri(ai,a;) ^ ^ e 7ri(a,a:) _ g — 7ri(a,sB) j _ _g(^ x y 

Since Oj is an arbitrary simple root, g(x) is antisymmetric with respect to W. Then g(x) is 
a sum of antisymmetric orbit functions. Representing n«>o( eir '^ — e _7ri ^ a,a: ') in the form (|4.3p . 
we see that in this form there exists only one term e*i{\,x) with a strictly dominant weight A 
and this weight coincides with p. Therefore, g(x) = ip p [x). Proposition is proved. ■ 

An invariant measure of the compact Lie group G, associated with the Weyl group W, 
is expressed in terms of \<p p \ 2 - Taking into account an explicit form of positive roots in the 
orthogonal coordinate systems, it is easy to derive from (j5.9j) that in these coordinates we have 



<p p {x) = (2i)^ +1 V 2 J] wntfa-Xj) forA n , 

l<i<j<n+l 

<p p (x) = (2i) n smir(xi — Xj) s'mir(xi + Xj) sin7TXj for B n , 

l<i<j<n l<i<n 

<p p (x) = (2i) n smir(xi — Xj) s'mir(xi + xj) sin27TXj for C n , 

l<i<j<n l<i<n 

<Pp(x) = (2i) n ^ n_1 ^ sin-7r(xj — Xj) sin7r(xj + Xj) for D n . 

l<i<j<n 

These formulas give other expressions for |<^p(x)| with respect to formulas derived in Subsec- 
tions IPH4TT1 

Proposition 9. The antisymmetric orbit function (p p vanishes on the boundary of the funda- 
mental domain F. It does not vanish on intrinsic points of F. 

Proof. Since p G , then ip p vanishes on the boundary of F due to the results of Subsec- 
tion [5]3j From the other side, it is easy to see from (|5.9[) that the set of points, on which (f p (x) 
vanishes, coincides with the set of all points x G E n for which (a, x) G Z for some root a. No of 
these points is an intrinsic point of F. The proposition is proved. ■ 

From Proposition [9] and from the above formulas for (f p (x) we easily derive explicit forms of 
the fundamental domains for the cases A n , B n , C n , D n in the orthogonal coordinates. 

(a) The fundamental domain F{A n ) is contained in the domain of real points x = (xi,x%, . . . , 
x n +i) such that 

XI > X2 > ■ ■ ■ > X n+ 1, XI + X2 H h x n+ i = 0. 

Moreover, a point x of this domain belongs to F(A n ) if and only if x\ + |x n +i| < 1. The last 
condition means in fact the relation (x, <Z 1 for the highest root £ of the root system A n . 

(b) The fundamental domain F{B n ) is contained in the domain of points x = (xi,X2, ■ ■ ■ , 
x n ) such that 

1 > X\ > X2 > ■ ■ ■ > X n > 0. 
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Moreover, a point x of this domain belongs to F(B n ) if and only if x\ + x 2 < 1. 

(c) The fundamental domain F(C n ) consists of all points x = (x\,x 2 , ■ ■ ■ ,x n ) such that 

\ > Xl > X2 > ■ ■ ■ > x n > 0. 

(d) The fundamental domain F(D n ) is contained in the domain of points x = (xi,x 2 , • • • , 
x n ) such that 

1 > Xi > X 2 > ■ ■ ■ > X n -\ > \ x n\- 

Moreover, a point x of this domain belongs to F{D n ) if and only if x\ + x 2 < 1. 
5.10 Symmetric orbit functions <p p 

Let us consider a symmetric counterpart of the antisymmetric orbit function <p p (x): 

M*) ■= E e 2 ^ x \ (5.10) 

where, as before, p = ^ a. 

Proposition 10. The symmetric orbit function (j) p can be represented as 



tp p ( x ) = Y[ (e ni{a > x) + e ~^ x) ) = e 2n ' l{p ' x) JJ (1 + e 



-2iri(a,x) ^ 

Q>0 Q>0 

= 2 r Yl cos ir(a,x), (5.11) 

where r is a number of positive roots in the corresponding root system. 
This proposition is proved in the same way as Proposition [6l 

It is easy to derive from (|5.1ip that in the orthogonal coordinate systems we have 
<f> p (x) = 2 n{ - n+1 ^ 2 cos ir(xi - Xj ) for A n , 

l<i<j<n+l 

4> p (x) = 2 n cos7r(xj — Xj) cosir(xi + xj) cosirxi for B n , 

l<i<j'<n l<i<n 

4> p (x) = 2 n cos7r(xj — Xj) cos7r(:Ej + Xj) cos27TXj for C n , 

l<i<j'<n l<i<n 

4> p (x) = 2 n ( n_1 ) cos7r(xj — Xj) cos7r(xj + Xj) for D n . 

l<i<j<n 

6 Properties of antisymmetric orbit functions of A n 

By using results on decomposition of certain reducible representations of the group GL(n, C) 
into irreducible constituents, properties of antisymmetric orbit functions of A n can be derived. 
We use in this section the results of 1 30 1 . 
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6.1 Decomposition of symmetric powers of representations 

We need some formulas for decomposition of symmetric powers of finite dimensional irreducible 
representations of the group GL(ni,C) x GL(ri2,C). Let us first define symmetric powers of 
representations. 

Let T be a finite dimensional representation of a group G on a linear space X. It induces 
a representation on the space V m (X), which is a subspace of the space V(X) of polynomials on X, 
consisting of all homogeneous polynomials of power m. In order to determine this representation 
we note that the formula 

(Q(9)p)(x) = P (T(g' 1 )x), xeX, geG, 

gives a representation of G on V(X) which is denoted by Q. The subspace V m (X) is invariant 
with respect to the representation Q. The restriction of Q onto V m (X) is called an m-th 
symmetric power of the representation T and is denoted as o~ m (T). 

Let G = GL(ni,C) x GL(n 2 ,C), n\ < n<i- Then a finite dimensional irreducible representa- 
tion of G is given (in the orthogonal coordinate system) by 

(A|/x) = (rai,m 2 ,...,m ni | n, r 2 , . . . , r„ 2 ), 

where 



mi > m>2 > • ■ ■ > m ni , n > r 2 > • • • > r. 



!!■>■ 



We assume that m ni > and r n2 > 0. 

One can consider three representations of G = GL(n±, C) x GL(n2,C) on the space Wl nin2 (C) 
of n± x 7i2 complex matrices. They are given by the formulas 

T( gi , 9 2)X = 9l Xgl (6.1) 
T'(g 1 ,g2)X = g 1 Xg* 2 , (6.2) 
T"( gi ,g 2 )X = g 1 Xg 2 -\ (6.3) 

where the index t means a transposition and * means a transposition together with complex 
conjugation. Then for symmetric powers of these representations we have the following decom- 
positions into irreducible representations of G (a proof see in [30] ) : 

o- m {T) = (si,S2,...,s ni \si,s 2 ,...,s ni ,0,...,0), (6.4) 

Si=m 

a m {T')= ^ (si, s 2 , . . . , s ni | si, s 2j . • . , s ni ,0, ■■■ , 0), (6.5) 

0- m {T")= (si,S2, • • • ,Sr>i I 0, . . . ,0, -s ni , -s n2 , . . . , -si), (6.6) 

Ei s i= m 

where summations are over all si, S2, ■ ■ ■ , s ni such that 

si > S2 > • • • > s ni > 0, si + s 2 H h s m = m 

and the overbar means that a representation is anti- analytic. 

If m = n 2 = n, then the formulas (|6.ip - (|6.3p determine the following tensor product repre- 
sentations of G = GL(n,C) on the space 9Jt n (C) of complex n x n matrices: 



(T l ®T 1 ){g)X=gXg\ 



Antisymmetric Orbit Functions 



35 



{T l ®T l ){g)X=gXg\ 
(T 1 ®f 1 ){g)X = gXg- 1 , 

where T\ is the first fundamental representation (with highest weight (1,0, ...,0)) of G = 
GL(n,C), T\ and T\ are the complex conjugate and contragredient representations to the rep- 
resentation Ti, respectively. Then according to (|6.4p - (|6.6p we have 

Cm (6.7) 

m 

a m (T x ®f 1 ) = Y,T m ® f m , (6.8) 

m 

Cm (6.9) 



m 



where T m is the irreducible representation of the group GL(n, C) with highest weight m = 
(mi,m2, • • • , m n ), mi > m2 > ■ ■ ■ > m n > 0, and the summation is over those m for which 
mi + fni + • • • + m n = m. 

Replacing the space 9Jt n (C) by the subspaces of all symmetric or all antisymmetric matrices 
from 97l n (C) we obtain the following decompositions of symmetric powers of the irreducible 
representations of GL(n, C) with highest weights (2, 0, ... , 0) and (1, 1, 0, . . . , 0), respectively: 

o "m(r(2,0,...,0)) = ^7 1 (2mi,2m 2 ,...,2m n ), (6.10) 
m 

, to(2(1,1,O,... ) O)) = /) T (mi,mi,...,m k ,m k ), (6- 11 ) 
m 

where the summations are over those m = (mi, mi % . . . , m n ) for which mi > rri2 > • • • > m n > 
and mi + rri2 + • • • + m n = m. In the second formula n = 2k; if n = 2k + 1, then on the right 
hand side we have to replace 3~{mi,mi,...,m fc ,m fc ) by T ( mi, mi ,...,mfe,mfe,0) • 

6.2 Properties of antisymmetric orbit functions of A n 

We represent antisymmetric orbit functions of A n in the orthogonal coordinate system as in 
formula (|4.9p . Let A = (mi,m2, . . . , m n+ i) and A + r = (mi + r, m2 + r, . . . , m n+ i + r), where r 
is a fixed real number. If x = (xi, x%, . . . , x n +i), xi + X2 + ■ ■ ■ + x n+ i = 0, and w € W, then we 
have 

It follows from this equality that 

¥>A0*0 = VA+r^), 4>\(x) = 4>X+r(x) (6.12) 

where A = (mi, ni2, ■ ■ ■ ,m n+ i) is given in the orthogonal coordinate system. This means that 
instead of m;, i = 1, 2, . . . , n + 1, determined by formulas of Subsection 13.31 we may assume that 
mi, mi-, ■ ■■ i m n+i are integers such that mi > m-2 > • • • > m n+ i > 0. We adopt this assumption 
in this subsection. 

For simplicity we introduce the following notations: 

e 2 ™i= Vj , i = l,2,...,n + l. 
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In order to receive relations for antisymmetric orbit functions we have to take characters of 
representations on both sides of relations of the previous subsection and to substitute the ex- 
pression 



for characters (see Subsection 19. II below). 
The relation (|6.4[) gives the equality 



Xa m (T)(gi,92) = ^2 Xs(gi)xs'(g2) 

Ei mi=m 

for characters of representations, where s' = (si, S2, ■ ■ ■ , s ni , 0, . . . , 0) and the sum is such as 
in (|6.4p . Multiply both sides by t m and sum up over all non-negative integral values of m. We 
get 

oo oo 

J2 tm ^ m (T)(9i,92) = J2 tm E Xs(9i)xA 9 2). (6.13) 

m=0 m=0 J2m.i=m 

i 

Take this relation for g\ = diag(yi,y2, • • • i Dm) an d 92 = diag(zi, Z2, ■ ■ ■ , z n2 ). For such g\ and <?2 
the left hand side takes the form 



00 

t Xa m (T){9l, 92) = l^Vx V 2 ■••Vni z\ z 2 ' ' ' z m , 



m=0 m=0 

ni 

where the second summation on the right hand side is over integral ri and pj such that ^ r « + 

i=i 

"2 

^2 pj = m. Therefore, the relation (|6.13p can be written as 

oo ni ri2 

E Xs(9i)xA92) = X[]\(l-ty t z j y\ 

where g± = diag(yi,y 2 , • • • ,2/m), 92 = diag(zi,z 2 , ■ ■ • ,z m ) and \t\ < 1, \yi\ < 1, \zj\ < 1. 

Further, we represent characters in (|6.13|) in terms of antisymmetric orbit functions and set 
t = 1. As a result, we receive the following relation for antisymmetric orbit functions: 

ni 712 

j=lj=l m 

where y = (yi,y 2 , ■ ■ ■ , y ni ), z = (z±, Z2, ■ ■ • , z n2 ), and p± and /02 are half-sums of positive roots 
for A ni _x and A n2 _i, respectively. In particular, if ni = n 2 = n, then 

n 

cpp(y)ipp(z) Yl (l-ViZj)' 1 = ^2iPs+p(y)fs+ P (z). 

i,j=l m 

Applying to this relation the Cauchy lemma, which states that 

n 

det ((1 - yiZj)- l ) n i j=1 = ip p {y)ip p {z) ViZj)' 1 , 

M=l 
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we obtain the identity 

E <Px+ P (y)<Px+p(z) = det ((1 - ViZj)~ l ) n i j=l ■ 

A 

Now we use the relation (|6.10p for representations of the group GL(n, C) in order to derive 
the equality 

X(2,o,-,o),m(s0 = E * 2m (f)> geGL(n,C), 

|m|=m 

where on the left hand side is the character of the representation cr m (Tf2 o o)) horn (|6.10p . 
2m = (2mi, 2m<i, . . . , 2m n ), mi > m,2 > ■ ■ ■ > m n > 0, |m| = mi + m2 + • • • + m n . We derive 
from this equality that 

oo oo 

£ X(2,0,...,0),m(ff)< m = E E ^m(<7)- (6.15) 
m=0 m=0 |m|=m 

Setting g = diag(yi, i/2) • • • > J/n)> for the left hand side we have the expression 

oo 

E X(2,0,...,0),m(9)t m = II C 1 - ^i)" 1 ' 
m=0 

where \yi\ < 1 and |t| < 1. Now using the Weyl character formula one receives 

E^m +P (y)t |m| =tp P (y) J] (6-16) 

Substituting into (|6.16j) the expression for (f2m+p(y)i setting t = 1 and using the evident 
relation 

E «r< 2 • • • C" = (1 - ai) _1 (l - a ia2 )- 1 • • • (1 - aia 2 • • • a n )~\ 

m 

we get a non-trivial identity 

/ . \ yy 1 yi O 

2- ( i_ y 2 )(1 V) ... (1 ? . ?) =My) n c 1 -^) , 

(il,i 2 ,...,in) n ^1^2^ V y tn> l<i<j<n 

where summation is over all permutations i 2 , ■ ■ ■ , i n ) of the natural numbers 1,2, ... ,n and 
sign (ii, «2) ■ • ■ , in) means a sign of the permutation (ii,i 2 , ■ ■ ■ ,i n ). 
In the same way, from the decomposition (I6.11|) we derive 

EvWy)t |m| = My) II (i-te)" 1 , (6-17) 

where summation is over m = (mi, ■ ■ ■ , Trif-), k = [n/2], mi > m 2 > ••■ > > 0, 
rii = (mi, mi, m 2 , m 2 , • • • , m^, m^) (here one has to add at the end if n = 2k + 1). As above, 
we receive from here the identity 

/ . • \ 77 1 77 2 

_ signal, i 2 ,...,i n )y il y i2 ■■■yi n - 1 f \ n (1 \-i 

E u- v . v .)(i- v . v . v . v .)...(i- v . ..... j = Mv) 11 i^-ViVj) ■ 
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In order to derive other identities we note that for 8 = (61,62, ... , 6 n ) one has 
lim W+p (e M )=^(A + rf. 

v — *U 

Taking into account this relation and substituting y = (1, . . . , 1) and z = (1, . . . , 1) into (|6.14p . 
we derive a relation for the antisymmetric orbit functions tp p : 

£ p«(m + p s )<)(m + p r ) = { Z^\yJ' 4 S HPs)4XPr)> (6-18) 

|m|=m 

where r > s > 0, m = (mi, 7712, . . ■ , m s ), rh = (mi, mi, ■ ■ ■ , m s ,Q, . . . , 0), |m| = mi + m-2 + • • • + 
m s and pk = (k — 1, — 2, . . . , 1, 0). 

From the relations (16.16P and (16.17P one similarly obtains the identities 

, x (m + \n(n + l) - 1)! , . , 

sr^ , x (m + — 1) — 1)! , . , 

g y ' (m+rt= W-i)-i)J y ' w - (6 - 20) 

where m = (mi, m.2, . . . , m v ), m = (mi, mi, m2, m2, . . . , m„, m v ), ^ = [n/2]. 
From (|6.18p - (|6.20p we derive the relations 



^^(m + /) >W(m + fr )tlH = (1-t)— p« 

m 

2 ^(2m + p)*H = (1 - r"("+ lV Vp(p), 

m 

2 V? P (2m + p)*M = (1 - 0- n(n - 1)/2 ^(p)- 



Other decompositions of the previous subsection lead to new relations for antisymmetric orbit 
functions of A n . 



7 Decomposition of products of (anti) symmetric orbit functions 

The aim of this section is to derive how to decompose products of (anti)symmetric orbit functions 
into sums of (anti)symmetric orbit functions. Such operations are fulfilled by means of the 
corresponding decompositions of (signed) orbits. 



7.1 Products of symmetric and antisymmetric orbit functions 

Invariance of symmetric orbit functions <f)\ and anti-invariance of antisymmetric orbit func- 
tions (f \ with respect to the Weyl group W lead to the following statements: 

Proposition 11. (a) A product of symmetric orbit functions expands into a sum of symmetric 
orbit functions: 

4>x4>ti = y^,n v (j) u , (7.1) 

V 

where an integer n u shows how many times the orbit function <f> v is contained in the product </>a<?V- 
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(b) A product of antisymmetric orbit functions expands into a sum of symmetric orbit func- 
tions: 



where n u are positive or negative integral coefficients. 

(c) A product of symmetric and antisymmetric orbit functions expands into a sum of anti- 
symmetric orbit functions: 



where n v are positive or negative integral coefficients. 

Proof. In the case (b) we have 

ip x (wx)ip^(wx) = (det w) 2 ip x (x)ip^(x) = (px{x)(p^(x). 

Therefore, the product fxf^ is invariant with respect to W. Hence, it can be expanded into 
symmetric orbit functions (see Subsection 7.8 in pQ). Since antisymmetric orbit functions (fx 
are sums of exponential functions e 2m ( a > x ) with coefficients ±1, the coefficients n u in (|7.2p are 
integers. In the case (c) we have a similar situation. The case (a) is considered in pQ . ■ 

In order to fulfill expansions (a)-(c) in an explicit form it is necessary to fulfill the corre- 
sponding decompositions of products of (signed) orbit, considering usual (not signed) orbits as 
signed orbits in which to all points the sign "+" is assigned, and to take into account that 
multiplication of (singed) orbit functions are reduced to multiplication of exponential functions. 

A product O i (A) <g> O i (A / ) of two signed orbits ± (X) and O i (A / ) (one or two of them can 
be replaced by usual orbits, that is, to the corresponding points the sign "+" is assigned) is the 
set of all points of the form Ai + A2 (where Ai £ ± (X) and A2 £ O (A')) with a sign which is a 
product of signs of A and A'. Since a set of points Ai + A2 (without signs), Ai £ O(A), A2 £ O(X'), 
is invariant with respect to action of the corresponding Weyl group, each product of orbits is 
decomposable into a sum of orbits. Then it follows from assertions (a)-(c) of Proposition [TT] 
that, considering points Ai + A2 with signs, we obtain decomposition of O i (A) (g) ± (A / ) into 
usual orbits, where to each point a sign is assigned (the same sign for points of a fixed orbit). 
Moreover, a product of a signed orbit with a usual orbit decomposes into signed orbits (not into 
usual orbits). 

Under product of (signed) orbits we may receive an orbit with signed points in which the 
sign "— " corresponds to a dominant weight. This means that we obtain a signed orbit with 
opposite signs for its points. In this case we say that a product of (signed) orbits contains 
the corresponding signed orbit with sign "— " and denote it by — ± (/x). That is why in (17. 2\) 
and (|7.3p negative coefficients can appear. 

Example. Orbits of A\. If a £ E\ is strictly positive, then the signed orbit of this point ± (a) 
consists of two signed points a + and —a~. It is easy to see that for the product ± (a) (8) O^ib) 




(7.2) 



V 




(7.3) 



we have 



± {a) ® ± (b) = {a+, -aT} ® {&+, -b~} 

= {(a + 6)+, (-a - 6)+} U {(\a - b\)~ , (-|a - b\)~} 
= 0(a + b)U-0(\a-b\), 



where 0{a + b) and 0{\a — b\) are usual (not signed) orbits. Therefore, 



ip a (x)ip b (x) = 4> a+ b(x) - 4>\ a - b \(x). 
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Similarly, we have 

± (a) <g> 0(b) = ± (a + b) U ± (a - 6), a > b > 0, 
± {a) ® O(b) = O i (a + b) U -O i (6 - a), 6 > a > 0. 

Thus, 

tfaip^ipiix) = (p a +b(x) + tpa-b(x), a > b > 0, 

ip a (x)(j) b (x) = ip a+b (x) - (p\ a - b \ (x), 6 > a > 0. 

For the corresponding decompositions for O(a) ® 0(b) see [I]. 

Decomposition of products of orbits in higher dimension of the Euclidean space is not a simple 
task. In the next subsection we consider some general results on the decomposition. 

7.2 Products of symmetric and antisymmetric orbits 

Let 0(A) = {w\\w E W/W\} be a usual orbit and O (/x) = {wfi\w E W} be a signed orbit. 
Then 

O(A) ® ± (/i) = {(wA + w//i) doW | u> E VF/Wa, w' E 

= {(u-A+u>i/i) dctu ' 1 1 weW/Wx} u {(wA+w2/u) dct "' 2 1 weW/Wx} u • • • 
U {(u;A+u; s /i) dct ^ | weW/W x }, (7.4) 

where u;2, • • • , w s is the set of elements of W. Since a product of an orbit and a signed orbit 
decomposes into signed orbits, for decomposition of the product O(A) (8) O i (/i) into separate 
signed orbits it is necessary to take dominant elements from each term of the right hand side 
of ([731). That is, O(A) <g> ± (/i) is a union of the signed orbits, corresponding to points from 

D{{{wX + wifi) dctWl \w E W/W x }), D({(w\ + w 2 fi) dctW2 \w E W/W x }), 

D({(wX + w s n) dctWs | w E W/W x }), (7.5) 

where D{{{wX + Win) dct Wl | w E W/W x }) means the set of dominant signed elements in {(wX + 
Win) dDtw < \w E W/W x }. 

Proposition 12. The product 0(A)®0 ± (/i) consists only of signed orbits of the form ± (|k;A + 
E W/W x , where \w\ + n\ is a dominant weight of the orbit containing wX + fJ,. Moreover, 
each such orbit O^^wX + fi\), w E W/W x , except for those of them, for which \wX + fi\ lies on 
some wall of the dominant Weyl chamber, belongs to the product O(A) (%> O (p). 

Proof. For each dominant element wX + Wifj> from (|7.5p there exists an element w" E W such 
that w"(wX + Will) = w'X + /i. It means that wX + Wifi is of the form \w'X + fJ,\, w' E WfW\. It 
is clear that a sign of this dominant element is det Wiw" . Conversely, take any element wX + /i, 
it; E W/W x . It belongs (with some sign) to the product O(A) <g> ± (/i). This means that 
\wX + fi\ also belongs to this product. Therefore, the signed orbit O (\wX + fi\) is contained in 
O(A)0O ± (/i) with some coefficient if \wX + //| does not lie on some wall of the dominant Weyl 
chamber. This coefficient cannot vanish since if it vanishes, then in the product O(A) (8) ± (//) 
there are contained points of ± (|wA + /i|), but taken with an opposite signs. In this case there 
exists another element w'X + fi such that wX + [i = w"(w'X + /x). Since /i does not lie on a wall, 
this is not possible. Proposition is proved. ■ 
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It follows from Proposition [12] that for decomposition of the product 0(A) (8) O (//) into 
separate signed orbits we have to take all elements wX + //,«)£ W/W\, and to find the 
corresponding strictly dominant elements \wX + n\, w G W/W\. 

Corollary. For i/ie product 0(A) <g> ± (//) we /jawe 

0(A) 8) O i (/i) = Ul eW/l y/ w0±(|wA + ^' (7 - 6) 

where the prime means that the term with \w\ + /i| Zym<7 on a u>aZ/ mwsi 6e omitted, and e w is 
equal to +1 or —1. 

Proof. This corollary is similar to Proposition 4 in pQ. A proof is also similar. We only have 
to take into account that signed orbits can be contained in O(A) (8) ± (fi) with the sign "— ". 
The corollary is proved. ■ 

Note that in the case of product 0(A) O(fx), X G P + , (J, G P + , of usual orbits the orbits 
0{v) with multiplicities m u > 1 may appear in the decomposition . As we see from Corollary, 
all coefficients in the decomposition (|7.6h are modulo 1. The only problem which appears here 
is to find signs of the coefficients e v . 

According to Corollary we have 

<f>x(?)M x ) =J2 weW/Wx £wl P\^+d x ^ 
where summation is such as in (I7.6P and e w are equal to +1 or —1. 

Proposition 13. Let O(X), A / 0, be an orbit and let O i (//) be a signed orbit. If all elements 
wX + [i, w G W/W\, are dominant, then 

O(X) 8) ± (/x) = (J' O^A + M ), 

where the prime means that terms corresponding to wX + /i lying on a wall must be omitted. 

Proof. The statement of this proposition follows from the above corollary. ■ 

At the end of this subsection we formulate the following method for decomposition of products 
O(A) (%> O^^fi), which follows from statement of Proposition [12j On the first step we shift all 
points of the orbit O(A) by /i. As a result, we obtain the set of points wX + fi, w G W/W\. On 
the second step, we map non-dominant elements of this set by elements of the Weyl group W 
to the dominant Weyl chamber. On this step we obtain the set \wX + fJ,\, w G W/W\. Then 
according to Proposition 1121 0(A) <8> ± (/i) consists of the signed orbits O^QwA + fi\) for which 
\wX + /i| do not lie on a wall of the dominant Weyl chamber. On the third step, we determine 
signs of these orbits, taking into account the above propositions or making a direct calculation. 

7.3 Products of antisymmetric orbits 

Let O i (A) = {(wX) dctw \w G W} and ± (^) = {(wn) dctw \w G W} be two signed orbits, where 
A and \i are strictly dominant elements of E n . Then 

O i (A) O i (/i) = {(wX + w'fj,) detww ' | w G W, w' G W} 

= {(wX+wui) dctwWl | weW} U {{wX+w 2 fi) dctwW2 \weW} U • • • 

U {(wX+w s fi) detww ° | weW}, (7.7) 
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where w\, W2, ■ ■ ■ , w a is the set of all elements of W. Since a product of two signed orbits 
decomposes into usual orbits (all points of some of these orbits are taken with the sign "— "), 
for decomposition of the product ± (A) &) ± (/u) into separate orbits it is necessary to take 
dominant elements from each term of the right hand side of (|7.7p . That is, ± (A) &) =t (//) is a 
union of the orbits (some of them with the sign "— ") corresponding to points from 

D({(w\ + wiv) detwwl \weW}), D({(w\ + w 2 ») detwW2 \w£W}), 

D({(wX + w sf i) dctwWs \w G W}), (7.8) 
where D({(wX + Wifi) detwWi \w G W}) means the set of dominant signed elements in {(wX + 

Win) dDtww *\w g W}. 

Proposition 14. The product ± (X) (8) =t (//) consists only of orbits (with signs "+" or "— ") 
of the form 0(\wX + n\), w G W , where \wX + fi\ is a dominant weight of the orbit containing 
u>A + /i. Moreover, each such orbit 0(\wX + fi\), w G W, for which \wX + [i\ does not lie on some 
wall of the dominant Weyl chamber, belongs to the product 0^(X) (g> ± (//). 

Proof. For each dominant element wX + Wifi from (|7.8p there exists an element w" G W such 
that w"(wX + Wifi) = w'X + fi. It means that wX + Wifi is of the form \w'X w' € W, and 
the first part of the proposition follows. 

Take any element wX + fi, w G W, which does not lie on a wall. Then \wX + fi\ does not lie 
on a wall. Then wX + [i belongs (with some sign) to the product ± (A) <g> ± (fi). This means 
that \wX + (i\ also belongs to this product. Therefore, the orbit 0(\wX + is contained in 
± (A) ® ± (fi) with some coefficient. This coefficient cannot vanish since if it vanishes, then 
in the product O i (A) (£> ± (//) there are contained points of O i (|w;A + /u|), but taken with an 
opposite signs. In this case there exists another element w'X + fi such that wX + fi = w" \w' X + /i). 
Since [i does not lie on a wall, this is not possible. Proposition is proved. ■ 

It follows from Proposition [TH that for decomposition of the product ± (X) ® ± (^i) into 
orbits we have to take all elements wX + /1, w G W, and to find the corresponding dominant 
elements \wX + fj,\, w G W. 

Corollary. For the product ± (X) (%> ± (/x) we have 

± (X) ± ( l i) = [J e w O{\wX + M |), (7.9) 

where e w is equal to +1 or —1 if \wX + /u| does not lie on some wall. 

In the case of product ± (A)®0 ± (/i), A G JPjT, /U G PjjT, of sighed orbits in the decomposition 
may appear orbits 0(v) with integral coefficients m u such that > 1. Such coefficients may 
appear only for v lying on a wall. 

Proposition 15. Let O ziz (X) and ± (fi) be two signed orbits. If all elements wX + n, w G W, 
are dominant, then 

O ± (A)0O ± (^)= J e. a ,0(u»A + ^), 

where e w is equal to +1 or —1, i/ |iuA + /i| does not lie on some wall. 

This proposition is proved in the same way as Proposition Q3] and we omit it. 
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Proposition 16. Let O (A) and O (/i) be sighed orbits, and let all elements wX + \i, w € W , 
be strictly dominant (that is, they are dominant and do not belong to any wall of the dominant 
Weyl chamber). Then 

± {\)®0 ± (ii)= [j e w O(w\ + fi), 

where e w are equal to +1 or —1. 

This proposition is proved in the same way as Proposition 2 in pQ. 

7.4 Decomposition of products for rank 2 

We give here examples of decompositions of products of (signed) orbits for the cases A% and C 2 . 
Orbits for these cases are placed on the plane. Therefore, decompositions can be done by 
geometrical calculations on this plane. These cases can be easily considered by using for orbit 
points the orthogonal coordinates from Section OH The corresponding Weyl groups have a simple 
description in these coordinates and this gives a possibility to make calculations in a simple 
manner. 

For the case of A 2 at a 7^ b we have 



A 2 : ± (a 


b) ® 0(c 


0) 


= ± (a+c b) U ± (a-c b+c) 










U -0 ± (a+b-c c-b) 


(a> c> b), 


± {a 


b) ® 0(c 


0) 


= ± {a+c b) U ± {a-c b+c) 










UO ± {a b-c) 


(a > c, b > c), 


± (a 


b) &> 0(c 


0) 


= ± (a+c b) 


(a = b = c), 


± {a 


b) ® 0(c 


0) 


= ± {a+c b) U -0 ± (a+b-c c-b)) 


(a = c > b), 


± (a 


b) &> 0(c 


0) 


= ± {a+c b)UO ± (a b-c) 


(b > a = c), 


± {a 


b) ® 0(c 


0) 


= ± {a+c b) U -0 ± (c-a a+b) 


(a < b = c), 


± (a 


b) &> 0(c 


0) 


= ± {a+c b) U -0 ± (c-a a+b) 










U ± (c-a-b a) 


(c > a + b), 


± (a 


b) ® 0(c 


0) 


= ± {a+c b) U -0 ± (c-a a+b) 










U -0 ± {a+b-c c-a) 


(a + b> c> b) 


± (a 


b) ® 0(c 


0) 


= ± (a+c b) U -0 ± (c-a a+b) 










UO ± {a b-c) 


(a + b> b> c) 



If a = 6, then we get 

± {a a) ® 0(c 0) = ± (a+c a) U O i (c-2a a) 

U -0 ± (c-a 2a) (c > 2a), 

± (a a) (8) 0(c 0) = ± (a+c a) U -0 ± (2a-c c-a) 

U -O i (c-a 2a) (2a > c> a), 

± (a a) (8) 0(c 0) = ± (a+c a) U ± (a-c a+c) 

UO ± (aa-c) (a>c). 

Similar products of Ci orbits are of the form 

C 2 : ± (a b) (8) 0(c 0) = ± {a+c b) U -O i (-a-26+c b) U -0 ± (c-a a+6) 

U ± (c-2b-a a+b) {a+b-c<b), 
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± {a b) <g) 0(c 0) = ± (a+c b) U ± {a+2b-c c-a-b) U ± {a-c b+c) 

U -0 ± (a+2b-c c-b) {b> c-a-b, a>c), 
± {a b) <g> 0(c 0) = ± {a+c b) U ± (a+26-c c-a-b) U -O i (c-a a+6) 

U -0 ± (a+2b-c c-b) {b> c-a-b, Oa), 
± (a b) <g> 0(c 0) = ± (a+c 6) U ± (a-c 6) U ± (a-c 6+c) 

U -O i (a+26-c c-b) (a+b>b+c), 
± (a b) ® 0(c 0) = ± (a+c 6) U -0 ± (c-o a+6-c) U -0 ± (c-a a+b) 

U -O i (a+26-c c-6) (&+c>a+&>c-&), 
± (a b) <g> 0(c 0) = ± (a+c 6) U -O i (c-a a+b-c) U -O i (c-a a+6) 

U ± (c-a-2b a+b) (a+b<c-b). 



The corresponding decompositions of products of antisymmetric and symmetric orbit func- 
tions can also be easily written down. 

8 Decomposition of antisymmetric VF-orbit functions 
into antisymmetric W-orbit functions 

As in Section[71 for these decompositions it is enough to obtain the corresponding decompositions 
for signed orbits. For this reason, we shall deal mainly with signed orbits. Our reasoning here 
is very similar to that of Section 4 in pQ. 

8.1 Introduction 

Let R be a root system with a Weyl group W, and let R' be another root system which is 
a subset of the set R. Then a Weyl group W' for R' can be considered as a subgroup of W . 

Let O w (X) be a signed W-orbit. The set of points of the usual orbit Ow(^) is invariant 
with respect to W'. This means that the signed orbit O^(X) consists of signed VF'-orbits. In 
this section we deal with representing O^(A) as a union of signed W'-orbits. Properties of 
such a representation depend on root systems R and R' (or on Weyl groups W and W). We 
distinguish two cases: 

Case 1: Root systems R and R' span vector spaces of the same dimension. In this case Weyl 
chambers for W are smaller than Weyl chambers for W'. Moreover, each Weyl chamber for W' 
consists of \ W/W'\ chambers for W. Let D + be a dominant Weyl chamber for the root system R. 
Then a dominant Weyl chamber for W' consists of W-chambers WiD + , i = 1,2, ... ,k, k = 
\W/W'\, where Wi, i = 1,2, ... ,k, are representatives of cosets in W/W'. If A does not lie on 
any wall of the dominant Weyl chamber D + , then 



where O w , are signed W^'-orbits. (Note that if det Wi = —1, then (detwi) O w ,(wi\) means the 
signed orbit 0^,(wi A) in which each point is taken with opposite sign.) 

Representing A by coordinates in w-basis it is necessary to take into account that coordinates 
of the same point in w-bases related to the root systems R and R' are different. There exist 
matrices connecting coordinates in these different w-bases (see [32]). 




(8.1) 



Antisymmetric Orbit Functions 



45 



For expanding an antisymmetric VF-orbit function into antisymmetric VF'-orbit functions it 
is necessary to take into account the formula (|8.1|) . Namely, the following expansion 

k 

^(x) = ^(det^)^' A (x) 
i=l 

corresponds to the decomposition (|8.ip . 

Case 2: Root systems R and R' span vector spaces of different dimensions. This case is more 
complicated. In order to represent O w (X) as a union of signed VF'-orbits, it is necessary to 
project points fx of O w (X) to the vector subspace E n > spanned by R' and to select in the set 
of these projected points dominant points with respect to the root system R'. Note that under 
projection, different points of (9^>(A) can give the same point in E n >. This leads to appearing 
of coinciding signed VF'-orbits in a representation of O w (\) as a union of VF'-orbits. Moreover, 
for some signed W-orbits their points must be taken with opposite signs. 

As in the previous case, under expansion of an antisymmetric VF-orbit function y>\(x) into 
antisymmetric VF'-orbit functions we have to consider <p\(x) on the subspace E n > C E n and to 
take into account the corresponding decomposition of the signed orbit O w (\). For this reason, 
below in this section we consider decomposition of signed PF-orbits into VF'-orbits. They uniquely 
determine the corresponding expansions for antisymmetric orbit functions. 

8.2 Decomposition of signed W^-orbits into Wa ti _ 1 -orbits 

For such decomposition it is convenient to represent orbit elements in orthogonal coordinates 
(see Section [3]). Let ± (A) = ± (mi, 777.2, • • • > mn+i) be a signed W^-orbit with dominant 
element A = (777.1,777.2, ■ ■ . , m n +i), where 

777,1 > 7772 >••• > TTl n > 777 n+ i. 

The orthogonal coordinates mi, 7772, . . . , m n+ i satisfy the conditions mi + 7712 + • • • + m n+ i = 0. 
However, we may add to all coordinates mi the same real number, since under this procedure 
the ^-coordinates Aj = m 8 — m, + i, i = 1, 2, . . . , 77 do not change (see Section [3|). 
The signed orbit ± (A) consists of all points 

10(7771,777.2, • . . ,m n+ i) = (m h ,m i2 , . . . ,m in+1 ), w £ W An , (8.2) 

where (11,12, ■ ■ ■ , in+i) is a permutation of the numbers 1, 2, . . . , 77 + 1, determined by w. The 
sign of (detw) is attached to such point. Points of O i (A) belong to the vector space E n+ \. We 
restrict these points to the vector subspace E n , spanned by the simple roots 01,02, ■ . . ,ot n -\ 
of A n , which form a set of simple roots of A n —\. This restriction is reduced to removing the 
last coordinate mj n+1 in points (777^,777^, . . . ,mj n+1 ) of the signed orbit O i (A) (see (|8.2p ). As 
a result, we obtain a set of points 

(m !l ,m i2 ,...,m i J (8.3) 

received from the points (|8.2[> . The point (|8.3I) is dominant if and only if 

m L > m i2 > ■ ■ ■ > m i n ■ 

It is easy to see that after restriction to E n (that is, under removing the last coordinate) we 
obtain from the set of points (18. 2p the following set of dominant elements: 

(mi, . . . ,777,-1, rhi,m i+ i, . . .,m n+1 ), i = 1,2, ... ,77+ 1, 

where a hat over m^ means that the coordinate m.; must be omitted. 
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Thus, the signed WA n -orbit ± (mi,m2, . . . ,m n +i) consists of the following signed Wa„_ 1 - 
orbits: 

± (mi, . . . ,mi-i,7hi,m i+1 , . . . ,m„ + i), i = 1, 2, . . . ,n + 1. 

Each of these signed orbits must be taken with a coefficient +1 or —1. Moreover, a coefficient 
at the orbit O (mi, . . . , m,_i, rhi, mj+i, . . . , m n +i) is 1, if after rhi in the point 

(mi, . . . ,mi-i,rhi,mi + i, . . . ,m n+ i) 

a number of coordinates is even, and — 1 otherwise. This statement completely determines an 
expansion of the antisymmetric orbit function y^^ 4 ^ m ^ into antisymmetric W^ Jl _ 1 -orbit 
function: 

ra+1 

V(ffH,m ai ...,m 1 » + i)( a = X]( detu, ( m »))^(™i,-.., mi -i,m,m !+ i,..,™„ + i)( 3; ). 
i=l 

where w(rrii) is the permutation which sends the coordinate m; to the end, not changing an 
order of other coordinates. 

8.3 Decomposition of signed Wa„_! -orbits 
into W Ap _ 1 X WAq_ x -orbits, p + q = n 

Again we use orthogonal coordinates for orbit elements. We take in the system of simple roots 
a±, a.2, • • • , a n -i of A n _i two parts as a±, ct2, • • • , a p -i and a p +i, ct P +2) • • • , <%p+q-i = ctn-\- The 
first part determines Wa p _ 1 and the second part generates W^_ r We consider a signed W^„_i- 
orbit (^(A), where A = (mi, m2, ■ ■ ■ , m n ), mi > rri2 > • • • > m n . This orbit consists of points 

w(mi,m2,...,ra„) = K,m !2 ,...,miJ, weW^, (8.4) 

where (ii,t2, ...,%) is a permutation of the numbers 1, 2, . . . , n. We restrict points (|8.4p to the 
vector subspace Ej, x S g spanned by the simple roots ai, «2, • • • > and a p +i, a p +2, • • • , otn-i, 
respectively. Under restriction the point (|8.4p turns into the point 

(m h ,m i2 ,..., m ip ) (m ip+1 , m ip+1 , . . . , m in ) . 

In order to determine a set of signed Wa p ^ 1 x WOi ^-orbits contained in the signed orbit ± (A) 
we have to choose from (|8.4p all elements for which 

m h > m i2 > ■ ■ ■ > m ip , m ip+1 > m ip+2 > ■■■ > m in . 

To find this set of points we have to take all subsets m^ , mj 2 , . . . , rrii p in the set mi, ttt-2, . . . , m n 
such that TTii x > rrii 2 > • • • > m^. Let S denote the collection of such subsets. Then (^(A) 
consists of signed Wa p _ 1 X WA q ^-orbits 

± ((m il ,m i2 , . . . ,m ip )(m jl ,m j2 , . . . ,m jq )), {m h ,m i2 , . . . ,m ip ) G S, (8.5) 

where (mj 1 ,mj 2 , . . . ,rrij ) is a supplement of the subset (m^, m.j 2 , . . . , m^ ) in me whole set 
(mi, ttt-2, . . . , m n ), taken in such an order that mj 1 > rrij 2 > • • • > mj 9 . Each of these WOi _ 1 x 
14 7 J 4 g _ 1 -orbits is contained in O (A) only once. Each such a signed orbit is contained in the 
signed orbit ± (A) with sign "+" if the set of numbers 

(m h , m i2 , . . . , m ip , m^ ,m h ,..., m jq ) 

from (|8.5p is obtained from the set (1,2, ...,n) by an even permutation and with sign "— " 

otherwise. The corresponding expansions of antisymmetric orbit functions </?^ An_1 ^(x) into 
antisymmetric Wa p _ 1 x Wa^j -orbit functions is evident. 
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8.4 Decomposition of signed Ws n -orbits into Ws n _ l -orbits 
and of signed Wc„-orbits into Wc n _ t -orbits 

Decomposition of signed W# n -orbits and decomposition of signed Wc n -orbits are fulfilled in the 
same way. For this reason, we give a proof only for the case of signed Wc n -orbits. 

A set of simple roots of C n consists of roots a%, cti, • • • , ct n . The roots a2, ■ ■ ■ ,a n constitute 
a set of simple roots of C n -\. They span the subspace E n -\. 

For determining elements A of E n we use orthogonal coordinates mi,ro2, . . . , m n . Then A is 
strictly dominant if and only if 

m\> ni2> • ■ ■ > m n > 0. 

Then the signed orbit ± (X) consists of all points 

w(m 1 ,rn2,-.-,m n ) = (±m il ,±rni 2 ,...,±rn in ), w £ W Cn , (8.6) 

where 12, ■ ■ ■ , in) is a permutation of the set 1,2, ... ,n, and all combinations of signs are 
possible. 

Restriction of elements (|8.6p to the vector subspace E n —i, defined above, reduces to removing 
the first coordinate im^ in (|8.6j) . As a result, we obtain from the set of points (|8.6j) the collection 

(±m i2 ,±m i3 ,...,±m in ), w G W Cn - 

Only the points (mj 2 ,m, 3 , . . . ,rrii n _ l ,mi n ) with positive coordinates may be dominant. More- 
over, such a point is dominant if and only if 

m i2 > m i3 > ■■■ > m in . 

Therefore, under restriction of points <\8.6\) to E n -\ we obtain the following strictly Wc n _ 1 - 
dominant elements: 

(mi,m2,...,mi-i,m ! ,m i+ i 1 ...,m n ), i = l,2,...,n, (8.7) 

where a hat over m^ means that the coordinate mi must be omitted. Moreover, the element (j8.7|) 
with fixed i can be obtained from two elements in (|8.6p . namely, from (mi, m<2, ■ ■ ■ , mj_i, ±mj, 
m, + i, . . . , m n ). In the signed orbit ± (mi,m2, . . . , m n ) these two elements have opposite signs. 

Thus, the signed Wc n -orbits O (mi,m2, ■ ■ ■ ,m n ) consists of the following signed Wc n - X - 
orbits: 

± (m 1 ,m 2 , . . . ,mi-i,rhi,m i+ i, . . . ,m n ), i = 1, 2, . . . , n. 

Each such signed M / c n _ 1 -orbit is contained in (mi, 777-2, ■ • • ,m n ) twice with opposite signs. 

Therefore, a restriction of the antisymmetric orbit function <P( mil m 2i ... jmn ) to the subspa- 
ce E n _\, described above, vanishes. 

For Wb k -orbits we have similar assertions. A signed Ws n -orbits ± (mi, ra%, . . . , m n ), mi > 
m,2 > • • • > m n > 0, consists of Ws Tl _ 1 -orbits 

± (m 1 ,m 2 , . . . ,mi-i,rhi,m i+ i, . . .,m n ), i = 1,2, . . . ,n, 

and each such orbit is contained in the decomposition two times (with opposite signs), that 
is, a restriction of the antisymmetric orbit function <P( OTl ,m2,— ,m„) °f B Tl to the subspace E n _\, 
described above, vanishes. 
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8.5 Decomposition of signed Wc„-orbits into W^_ 1 X Wc,j- or bits, p + q = n 

If a,\,a,2, . . . ,a n are simple roots for C n , then a±, a%, . . . , a p -\ are simple roots for A p -\ (they 
can be embedded into the linear subspace E p ) and a p+ i,a p+ 2, ... ,a n are simple roots for C q 
(they generate the linear subspace E q ). 

We use orthogonal coordinates for elements of E n and consider a sig ned W Cn -orbit ± (A), 
mi > m2 > • • • > m n > 0. This orbit consists of all points (|8.6p . Restriction of these points to 
the vector subspace E p x E q reduces to splitting of coordinates (|8.6j) into two parts: 

(±m il ,±m i2 ,..., ±m ip ) (±m ip+1 ±m in ) . (8.8) 

Due to the condition m\ > ni2 > • • • > m n > 0, these elements do not lie on walls of the 
[ x VFc^-chambers. We have to choose dominant elements (with respect to the Weyl group 
Wa„_i x Wc q ) in the set of points (|8.8p . The conditions of dominantness for elements of E p 
and E q show that only the elements 

(ma,-- ■ , m ij,-m ij+1 ■ ■ ■ ,-m p )(m p+1 ,-- -,m in ), j = 0, 1,2, . . . ,p, 
satisfying the conditions 

m h > m i2 > ■■■ > m iv m ij+1 < m ij+2 < ■■■ <m ip , rn ip+1 > m p+2 > • • • > m iri , 

are dominant. Moreover, each such point is contained in the signed VFo„-orbit ± (A) only once. 
This assertion completely determines a list of signed Wa p _ 1 x Wc 9 -orbits in O i (A). Each signed 
l^A p _i x Wc, -orbit is contained in O i (A) only once. 

This assertion uniquely determines a list of antisymmetric Wa t x VFc 9 -orbit functions, 
contained in the antisymmetric Wc n -orbit function <p\. However, it is necessary to determine 
signs of antisymmetric Wa p _ 1 X Wc? -orbit functions in the decomposition. It is easily made by 
using the description of the Weyl groups in the Euclidean space E n with orthogonal coordinates. 

8.6 Decomposition of signed Wo^-orbits into signed Wr> n ^-orbits 

Assume that a±, a<2, . . . , a n is the set of simple roots of D n , n > 4. Then a.2, ■ ■ ■ , a n are simple 
roots of D n _i. The last roots span the subspace E n -\. 

For elements A of E n we again use orthogonal coordinates m\,m2, • • • , m n . Then A is strictly 
dominant if and only if m\ > m2 > • • • > fn n _i > \m n \. We assume that A satisfies the condition 

mi > 77i2 > • • • > m n > 0. 

Then the signed orbit ± (X) consists of all points 

w(mi,m2,.--,m n ) = (±m il ,±mi 2 ,...,±m in ), w£W Dn , (8.9) 

where (ii,i2, ■ ■ ■ , in) is a permutation of the numbers 1,2, ... , re and there exists an even number 
of signs "— ". Restriction of elements (|8.9p to the subspace E n -\ reduces to removing the first 
coordinate im^ in (j8.9|) . As a result, we obtain from the set of points (18. 9p the collection 

(±m i2 ,±m i3 ,...,±m in ), w£W Dn , 

where a number of signs "— " may be either even or odd. Only points of the form (rrej 2 , , . . . , 
77ij n _ 1 , ±mi n ) may be dominant. Moreover, such a point is dominant if and only if 

m i2 > m i3 > ■ ■ ■ > m in _ 1 > \m in \. 
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Therefore, under restriction of points (|8.9p to £? n -l we obtain the following W£) n _ 1 -dominant 
elements: 

(mi,m2,...,mi-i,m i ,m i+ i,...,m re _i,±m n ), « = 1, 2, ...,n, (8.10) 

where a hat over rrij means that the coordinate mi must be omitted. Moreover, the ele- 
ment (|8.10p with fixed i can be obtained only from one element in (|8.9p . namely, from element 
(mi,m 2 , . . . , rrii-i, ±mj, mj+i, . . . , ±m n ), where at rrii and m n signs are coinciding. 

Thus, the signed Wn n -orbit O (mi,m 2 , . . . ,m n ) with mi > m 2 > • • • > m n > consists of 
the following signed WD n _ 1 -orbits: 

± (m x ,m 2 , . . . ,mi-i,rhi,m i+ i, . . . ,±m n ), i = 1,2, . . . ,n. 

Each such signed Wo n _ 1 -orbit is contained in O (mi, m 2 , ■ ■ ■ ,m n ) only once (with sign "+" or 
"— "). A sign of such an orbit depends on a number i and on a sign at m n . This sign is uniquely 
determined by the sign (detui) of the corresponding element w of Wo n - 
It is shown similarly that the signed Wo n - orbit 

± (mi, . . . ,m n _i, -m n ), mi > m 2 > ■ ■ ■ > m n > 

consists of the same signed Wjj n ^ 1 -orbits as the Wn n -orbits O (mi, m n _i, m n ) with the 
same numbers mi, . . . , m n -i,m n does. 

The above assertions uniquely determine expansions for the corresponding antisymmetric 
Wd„ -orbit functions. 

8.7 Decomposition of signed Wo^-orbits into Wa p _ 1 X W^-orbits, 
p-\-q = n, q>4 

If ai, a 2 , ■ ■ ■ , a, n are simple roots for D n , then ai, a 2 , . . . , a p -i are simple roots for A p _i (they 
can be embedded into the Euclidean subspace E p ) and a p+ i, a p+2 , . . . , a n are simple roots for D q 
(they generate the Euclidean subspace E q ). 

We use orthogonal coordinates in E n and consider signed W r o n -orbits O i (A) with A = 
(mi,m 2 , . . . , m n ) such that mi > m 2 > • • • > m n > 0. The orbit 0(A) consists of all points (|8.9p . 
Restriction of these points to the vector subspace E p _i x E q reduces to splitting the set of coor- 
dinates (|8.9p into two parts: 

(±m h ,±m i2 ,..., ±m ip ) (±m ip+1 ±m in ) . (8.11) 

Due to the condition mi > m 2 > • • • > m n > 0, these elements do not lie on walls of the 
Wa p -\ x H / £) 9 -chambers. We have to choose dominant elements (with respect to the Weyl group 
Wa„_ 1 x Wr) q ) in the set of points (|8.1ip . The conditions of dominantness for elements of E p 
and E q show that only the elements 

(m h ,...,mi j ,-m ij+1 . . . , -m ip )(m ip+1 , . . . , ±m in ), j = 0, 1, 2, . . . ,p, 

having even number of sign minus and satisfying the conditions 

m h > m i2 > ■ ■ ■ > m i:j , m ij+1 < m ij+2 < ■■■ <m ip , m ip+1 > m ip+2 > ■■■ > m in , 

are dominant. Moreover, each such point is contained in the W£j n -orbit O ziz (X) only once. These 
assertions completely determine a list of signed Wa p _ 1 x Wd 9 -orbits in the signed W£) n -orbit 
± (A). All signed W Ap ^ x Wz^-orbits are contained in O^(X) with multiplicity 1 and with 
sign "+" or "— ". This determines uniquely expansions for the corresponding antisymmetric 
Wo n -orbit functions. 
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9 Characters of representations and antisymmetric 
orbit functions 

Antisymmetric orbit functions (f\(x) with A G P^ are closely related to characters of irreducible 
representations of the corresponding compact Lie group G. This relation serves for derivations 
of some properties of antisymmetric orbit functions. 

9.1 Connection of characters with orbit functions 

To each Coxeter-Dynkin diagram there corresponds a connected compact semisimple Lie 
group G. Let us fix a Coxeter-Dynkin diagram and, therefore, a connected compact Lie group G. 
A complex valued function f(g) on G satisfying the condition 

f(g) = f(hgh- 1 ), h€G, 

is called a class function. It is constant on classes of conjugate elements. 

For simplicity, we assume that G is realized by matrices such that the set of its diagonal 
matrices constitute a Cartan subgroup, which will be denoted by H. This subgroup can be 
identified with the n-dimensional torus T, where n is a rank of the group G. The subgroup H 
can be represented as H = exp(irj), where f) is a real form of an appropriate Cartan subalgebra 
of the complex semisimple Lie algebra, determined by the Coxeter-Dynkin diagram. 

It is well-known that each element g of G is conjugate to some element of H, that is, class 
functions are uniquely determined by their values on H. 

There exists a one-to-one correspondence between irreducible unitary representations of the 
group G and integral highest weights A G P+, where P+ is determined by the Coxeter-Dynkin 
diagram. The irreducible representation, corresponding to a highest weight A, will be denoted 
by T\. The representation T\ and its properties are determined by its character Xx(g), which is 
defined as the trace of T\ (g) : 

Xx(g)=TrT x (g), g G G. 

Since Tr T\(/ig/i _1 ) = Tr T\(g), h G G, the character xx is a class function, that is, it is uniquely 
determined by its values on the subgroup H. 

All the operators T\(h), h G H, are diagonal with respect to an appropriate basis of the 
representation space (this basis is called a weight basis) and their diagonal matrix elements are 
of the form e 2m ^' x \ where /i G P is a weight of the representation T\, x = (xi,x%, . . . ,x n ) are 
coordinates of an element t of the Cartan subalgebra f) in an appropriate coordinate system 
(that is, coordinates on the torus T) and (■,■) is an appropriate bilinear form, which can be 
chosen coinciding with the scalar product on E n , considered above. Then the character xx(h) 
is a linear combination of the diagonal matrix elements: 

Xx(h)= Yl <e 2 -^>, (9.1) 

where -D(A) is the set of all weights of the irreducible representation T\ and is a multiplicity 
of the weight fi G D(\) in the representation T\. It is known from representation theory that 
the weight system D(\) of T\ is invariant with respect to the Weyl group W, corresponding to 
the Coxeter-Dynkin diagram, and = c^, w G W, for each fi G D(\). This means that the 
character xx{h) can be represented as 



Xx(h) = ^ c^(x), 

M6D + (A) 



(9.2) 
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where D + (A) is the set of all dominant weights in -D(A) and (j>n{x) is a symmetric orbit function, 
corresponding to the weight p G D+(A). Representing X\(h) as Xx( x )i where x = (xi,X2, ■ ■ ■ , x n ) 
are coordinates corresponding to the element f 6 I) such that h = exp2-7rii, we can make an 
analytic continuation of both sides of (|9.2p to the n-dimensional Euclidean space E n . Since the 
right hand side of (|9.2p is invariant under transformations from the affine Weyl group W aS , 
corresponding to the Weyl group W, the function x\( x ) 1S a ^ so invariant under the affine Weyl 
group W aS . That is, it is enough to define x\{ x ) only on the fundamental domain F of the 
group W aS . To this fundamental domain F there corresponds a fundamental domain (we denote 
it by F) in the subgroup H (and on the torus T). 

Many properties of orbit functions follow from properties of characters xa> an d we consider 
them as known from representation theory. 

The well-known Weyl formula for characters of irreducible representations of the group G 
states [33] that under appropriate selection of coordinates . . . ,x n ) of points h G H we 

have 

J2 (detw)e 2ni ^ + P' x ^ 
XX{ ' £ (det w)e^M <f P ( x ) ' 

where, as before, p = \ ^2 a - This is why antisymmetric orbit functions are so important for 

representation theory. Note that for dominant A G the element A + p is strictly dominant; 
for this reason, the antisymmetric orbit function (p\ +p (x) in (|9.3p does not vanish for A G P+. 



9.2 Orthogonality of characters 

The relation (|9.3p between characters of irreducible representations of compact Lie groups and 
the corresponding antisymmetric orbit functions leads to a simple proof of orthogonality of 
irreducible characters X\(h) = X\( x )- Indeed, due to the orthogonality (|5.2p for antisymmetric 
orbit functions and to the relation (19. 3p we have 

/ <p\+p(x)<px +p (x)dx = / x\{ x )X\'{ x )W P { x )?dx = 6\x>, 
Jf Jf 

that is, irreducible characters are orthogonal on F with respect to the measure \ip p (x)\ 2 dx. The 
expressions for the function (f p (x) for the classical compact Lie groups are given in Subsection l5.9[ 



9.3 Relations for antisymmetric orbit functions 

The formulas (|9.2p and (|9.3|) are a source of relations for antisymmetric orbit functions. 
Comparing the expressions (|9.2p and (|9.3p for characters, one gets the relation 

(f\+ p (x) = ^2 c%(f>n(x)<pp(x), (9.4) 
p,eD+(X) 

where, as before, are multiplicities of weights \i in the irreducible representation T\ of G. 

Let T\ and T p be two irreducible representations of G. Then their tensor product decomposes 
as a direct sum of irreducible representations of G as 



(9.5) 
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where mi^ is a multiplicity of the irreducible representation T u in the tensor product. Since T\ 
and Tfj, are finite dimensional representations, the sum on the right hand side of (|9.5|) is finite. 
To the decomposition (|9.5p there corresponds a relation for characters, 



Xx(x)x„(x) = Yl m ^Xu(x). (9.6) 
ueP+ 

Due to (19.31) it can be written as 



tp p (x) <p p (x) £g ^ (p p (x) 

where summation is such as in (|9.6p . Therefore, we have the following expression for a product 
of two antisymmetric orbit functions: 

ipx+p(x)fp +P (x) = ^2 mjf<p v+ p(x)<pp(x). (9.7) 
ueP + 

In particular, if fi = u>i, where Ui is i-th fundamental wight of the group G, then this formula 
takes the form 

cpx+p(x)(p Wl+p (x) = ^ m$*<p v+ p(x)(p p (x). 

Since multiplicities of irreducible constituents in the tensor product T\ % T Ui for many groups 
can be found in a simple way (in many cases these multiplicities are equal to 1; for example, if 
i = 1, then for the groups SU(n) and SO(n) they do not exceed 1), then this formula can be 
considered as a recurrence relation for antisymmetric orbit functions. 



10 Antisymmetric orbit function transforms 

As in the case of symmetric orbit functions, antisymmetric orbit functions determine certain 
orbit function transforms which generalize the sine transform (in the case of symmetric orbit 
functions these transforms generalize the cosine transform) |29} I3T] . 

As in the case of symmetric orbit functions, antisymmetric orbit functions determine three 
types of orbit function transforms: the first one is related to the antisymmetric orbit func- 
tions (f\(x) with integral A, the second one is related to <p\(x) with dominant A G E n , and the 
third one is the related discrete transforms. 

10.1 Decomposition in antisymmetric orbit functions on F 

Let f(g) be a continuous class function on G (see Subsection 19. ip . It defines a continuous 
function on the commutative subgroup H. We assume that this function on H has continuous 
partial derivatives of all orders with respect to analytic parameters on H. Such function / can 
be decomposed in characters of irreducible unitary representations of G: 

f(h) = ]T c x xx(h). (10.1) 

A6P+ 

We see from this decomposition that each class function is symmetric with respect to the corre- 
sponding affine Weyl group W aS (since characters xx admit this symmetry) and, therefore, is 
uniquely determined by its values on the fundamental domain F in H. 
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Going to the coordinate description of the points h £ H (see Subsection 19. ip . we obtain 

f( x ) = E c xXx( x ), x E E n . 
AeP+ 

Taking into account formula (|9.3p for the characters we receive 

^00/0*0 = E c \<p\+ P ( x )- ( io - 2 ) 

AGP+ 

Due to Proposition we may state that any antisymmetric (with respect to the affine Weyl 
group VF aff ) continuous function / on E n , which has continuous derivatives and vanishes on the 
boundary dF of the fundamental domain F, can be represented in the form f{x) = (p p {x)f(x), 
where f(x) is a symmetric (with respect to iy aff ) continuous function on E n with continuous 
derivatives. Thus, due to (|10.2j) we may state that any antisymmetric (with respect to W ) 
continuous function f on E n , which has continuous derivatives and vanishes on the boundary dF, 
can be decomposed in antisymmetric orbit functions (fx, A G P+, 

f{x) = Wx( x )- (10.3) 
x<=p+ 

By the orthogonality relation (|5.2p for antisymmetric orbit functions, the coefficients c\ in 
this decomposition are determined by the formula 



c A = / f(x)<p x {x)dx. (10.4) 



<F 

Moreover, the Plancherel formula 

E n 2 = / i/o*oi 2 «k ( io - 5 ) 

AeP + Jf 

holds, which means that the Hilbert spaces with the appropriate scalar products are isometric. 

Formula (|10.4p is the antisymmetrized Fourier transform of the function f(x). Formula (|10.3p 
gives an inverse transform. Formulas (|10.3p and (|10.4p give the orbit function transforms corre- 
sponding to antisymmetric orbit functions (p\, X £ Pj£. 

Let Cl(F) denote the Hilbert space of functions on the fundamental domain F, which vanish 
on the boundary dF of the fundamental domain, with the scalar product 



</l,/2> = / fl(x)f 2 (x)dx. 



F 



The set of continuous functions on F (vanishing on the boundary dF) with continuous derivatives 
is dense in Cq(F). Therefore, the formulas (|10.3p - (110.5p show that the set of orbit functions ip\, 
A £ , form an orthogonal basis of Cq(F). 



10.2 Symmetric and antisymmetric multivariate sine and cosine series 

Symmetric and antisymmetric orbit functions for the Coxeter-Dynkin diagram C n can be ex- 
pressed in terns of symmetric and antisymmetric multivariate sine and cosine functions (see 
formulas (|4.19p and (|5.4p ). Their application in the formulas for symmetric and antisymmetric 
orbit function transforms gives antisymmetric and symmetric multivariate sine and cosine series 
expansions. 
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The formulas f)10.3|) and (|10.4j) . applied to the case C n , determine expansions of functions, 
given on the fundamental domain 

F = {1/2 > x\ > x 2 > ■ ■ ■ > x n > 0} 

for the Coxeter-Dynkin diagram C n , into antisymmetric multivariate sine functions: 

f{x)= ^2 c m det(sm27rmiXj)^ j=1 , (10.6) 

meP{ 

where m = (m^mj, . . . , m n ) are integer n-tuples such that mi > 777.2 > • • • > m n > 0, and the 
coefficients c m are given by the formula 

c m = 2 2n / f(x) det (sin 27rm i x j )^. 1 dx. (10.7) 
Jf 

The Plancherel formula is of the form 



V |c m | 2 = 2 2 " / f(x)\f(x)\ 2 dx. 
Jf 



mSP^ 

Similarly, using the symmetric orbit function transform on the fundamental domain F (see 
Subsection 8.2 in PQ), determined by symmetric orbit functions for the Coxeter-Dynkin diag- 
ram C n given by the formulas (|4.24p and (|5.4p . one obtains symmetric multivariate cosine ex- 
pansion on F: 

f{x)= ^2 c m det + (cos27rmjXj)" J=1 , (10.8) 

mGP+ 

where m = (mi,m2, . . . , m n ) are integer n-tuples such that m\ > m2 > ■ ■ ■ > m n > 0, and the 
coefficients c m are given by the formula 



c m = 2 2n / f(x)det + {cos2Trm i x j )l j=1 dx. (10.9) 
if 

The Plancherel formula is of the form 



£ |Cm| 2 = 2 2 " jj{ X )\f{x)\ 2 dx. 



meP+ 



10.3 Orbit function transform on the dominant Weyl chamber 

The expansion (|10.3p of functions on the fundamental domain F is an expansion in the antisym- 
metric orbit functions (p\(x) with integral strictly dominant weights A. The antisymmetric orbit 
functions <f\(x) with A lying in the dominant Weyl chamber (and not obligatory integral) are 
not invariant with respect to the corresponding affine Weyl group W aS . They are invariant only 
with respect to the Weyl group W. A fundamental domain of W coincides with the dominant 
Weyl chamber D + . For this reason, the orbit functions (p\(x), A G Df, determine another orbit 
function transform (a transform on D + ). 

We began with the usual Fourier transforms on M n : 

/(A) = / f(x)e 2 ^dx, (10.10) 

f{x) = [ f(X)e- 2 ^d\. (10.11) 
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Let the function f(x) be anti-invariant with respect to the Weyl group W, that is, f(wx) = 
(det w)f(x), w £ W. It is easy to check that the function /(A) is also anti-invariant with respect 
to the Weyl group W. Replace in (jlO.lOp A by wX, w £ W, multiply both sides by detw;, and 
sum these both side over w £ W. Then instead of (jlO.lOp we obtain 

/(A) = f f(x)<p x (x)dx, A £ D+, (10.12) 



D 



+ 



where we have taken into account that f(x) is anti-invariant with respect to W. 
Similarly, starting from (llO.lip . we obtain the inverse formula: 



f{x) = / f(X)<p x (x)dX. (10.13) 

JD+ 

For the transforms (110.120 and (jlO. 13)) the Plancherel formula 



\f(x)\ 2 dx = / |/(A)|^A 

D+ JD+ 



holds. 



10.4 Symmetric and antisymmetric multivariate sine 
and cosine integral transforms 

The orbit function transforms (jl0.12f) and (jl0.13f) in the case of the Coxeter-Dynkin diagram C n 
lead to symmetric and antisymmetric multivariate sine and cosine integral transforms. 

Taking into account the expression (|4.19p for antisymmetric orbit functions for C n we obtain 
the transform 

/(A)= f f(x)det(sm2n\ i x j )l j=1 dx, (10.14) 

where A = (Ai, A2, . ■ ■ , A n ) £ D~\_, that is, Ai > A2 > • • • > A n > (the function /(A) vanishes 
on the boundary of D + ). The inverse transform is of the form 

f{x) = 2 2n f /(A) det (sin27rA^)" i= i dX. (10.15) 
For these transforms the Plancherel formula 



\f(x)\ 2 dx = 2 2n \f(X)\ 2 dX 

D+ JD+ 



holds. 

Similar transformations hold for symmetric multivariate cosine function: 

/(A)= ! /(x)det + (cos27rA i x J )^ =1 a!x, (10.16) 

J D+ 

f(x) = 2 2n f /(A)det+ (cos 2irX iXj )l j=1 dX, (10.17) 

J D-\- 

where A = (Ai, A2, • - - , A n ) E D+ and the function det + (cos XiXj)^. =1 is given by formula (|5.4p . 
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11 Multivariate generalization of the finite Fourier transform 
and of finite sine and cosine transforms 

Along with the integral Fourier transform there exists a discrete Fourier transform. Similarly, it 
is possible to introduce a finite orbit function transform, based on antisymmetric orbit functions. 
It is done in the same way as in the case of symmetric orbit functions in [1] by using the results 
of the paper [21] (see [31]). We first consider the finite Fourier transform. Then we consider a 
general theory (appropriate for any connected Coxeter-Dynkin diagram) . In the last subsections 
we give concrete antisymmetric and symmetric generalizations of the finite Fourier transform. 
In particular, we consider antisymmetric and symmetric multivariate finite Fourier transforms, 
discrete sine and cosine transforms, and antisymmetric and symmetric multivariate discrete sine 
and cosine transforms. 

11.1 Finite Fourier transform 

Let us fix a positive integer N and consider the numbers 

e mn ■= N~ 1/2 exp(2Trimn/N), m, n = 1, 2, . . . , N. (11.1) 
The matrix (e mn )m n =\ is unitary, that is, 

k k 

Indeed, according to the formula for a sum of a geometric progression we have 

t a + t a+l + ... + t a+r = ^ _ ty-lf^ _ f +1), t ^ ^ 

t a + t a+l + ■■■ +t a+r = r+l, t=l. 

Setting t = exp(27ri(m — n)/N), a = 1 and r = N — 1, we prove (|11.2|) . 

Let /(n) be a function of n £ {1, 2, . . . , N}. We may consider the transform 

N N 

f(n)e mn = N" 1 ' 2 f(n) exp(27rimn/A0 = f(m). (11.3) 

n=l n=l 

Then due to unitarity of the matrix (e mn )^ n=1 , we express f(n) as a linear combination of 
conjugates of the functions (jll.ip : 

N 

f(n) = N~ 1/2 Y f( m ) exp(-27rimn/iV). (11.4) 

m=l 

The function f{m) is a finite Fourier transform of /(n). This transform is a linear map. The 
formula (jll.4p gives an inverse transform. The Plancherel formula 

N N 
m=l n=l 

holds for transforms (jll.3p and (jll.4p . This means that the finite Fourier transform preserves 
the norm introduced in the space of functions on {1, 2, ... , N}. 

The finite Fourier transform on the r-dimensional linear space E r is defined similarly. We 
again fix a positive integer N . Let m = (mi, rri2, . . . , m r ) be an r-tuple of integers such that 
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each rrii runs over the integers 1,2, ... ,N. Then the finite Fourier transform on E r is given by 
the kernel 

e mn := e mini e m2n2 ■ ■ ■ e mrrir = N~ r/2 exp(27rim • n/N), 

where m • n = mini +m-2re2 + • • ■+m r n r . If F(m) is a function of r-tuples m, rrii £ {1,2, ... , N}, 
then the finite Fourier transform of F is given by 

F(n) = N" r / 2 F(m) exp(27rim • n/N). 

m 

The inverse transform is 

F(m) = N~ r/2 ^ F(n) exp(-2vrim • n/N). 

n 

The corresponding Plancherel formula is of the form ^ |.F(m)| 2 = ^ |F(n)| 2 . 

m n 

11.2 VF-invariant lattices 

In order to determine an analogue of the finite Fourier transform, based on antisymmetric orbit 
functions, we need an analogue of the set 

{m = {m 1 ,m 2 , . . .,m n } \ rm £ {1,2, . . . , N}}, 

used for multidimensional finite Fourier transform. Such a set has to be invariant with respect 
to the Weyl group W (see [2Tj). 

We know that Q v is a discrete P^-invariant subset of E n . Clearly, the set is also 

W- invariant, where m is a fixed positive integer. Then the set 

is finite and VF-invariant. If a\, a.%, . . . , ai is the set of simple root for the Weyl group W , then 
T m can be identified with the set of elements 

1 

m^ is ^dia( , di = 0,1,2, ... ,m — 1. (H-5) 

i=l 

We select from T m the set of elements which belongs to the fundamental domain F. These 
elements lie in the collection — Q v n F. 

Let /1 £ ^T be an element determining an element of T m and let M be the least positive 
integer such that Mfj, £ P v . Then there exists the least positive integer TV such that Nji G Q v . 
One has M\N and N\m. 

The collection of points of T m which belong to F (we denote the set of these points by Fm) 
coincides with the set of elements 

s = 17r^ + '" + 7J uj i ' w i=7 v. ( 1L6 ) 

M M (a.i,oti) 

where s\, s<2, . . . , S; runs over values from {0, 1, 2, ... } and satisfy the following condition: there 
exists a non-negative integer sq such that 



■so 



+ ^2simi = M, (11.7) 
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where mi, 772-2, • • • , m i are positive integers from formula (|2.8p . Values of mj for all simple Lie 

algebras can be found in Subsection 12.41 

Indeed, the fundamental domain consists of all points y from the dominant Weyl chamber for 

1 

which (y, £} < 1, where £ is the highest (long) root, £ = Yl m i a i- Since for elements s of (|11.6p 

i=l 

one has Si/M > and 

1 ' 1 

i=i 

then s € F. The converse reasoning shows that points of D i 7 must be of the form (jll.Gj) . 

To every positive integer M there corresponds the grid Fm of points (|11.6|) in F which 
corresponds to some set T n such that M\m. The precise relation between M and n can be 
defined by the grid Fm (see |2T] ). Acting upon the grid Fm by elements of the Weyl group W 
we obtain the whole set T m . 

Since antisymmetric orbit functions vanish on the boundary of a fundamental domain F, it 
make sense to consider also a subgrid F M consisting of all points of Fm which do not lie on 
a wall of F. 

Example. Grids Fm for A\. We take into account results of Example in Subsection l4.1[ For A\ 
we have w v = u = a/2, where a is the simple root. Elements of P+ coincide with mtv, m 6 Z + . 
Fixing M E Z + we have 

Fm = { s = m> where sq + si = M for sq, si € Z- } . 

Therefore, 

p — /n _L JL M-i 1 \ 
-CM — \u, M , M , , . . . , M 

11.3 Grids F M for A 2 , C 2 and G 2 

In this section we give some examples of grids Fm for the rank two cases (see |14| and [15j). 
Since the long root £ of A2 is representable in the form £ = a\ + «2, where ai and a 2 are simple 
roots, that is, m\ = m 2 = 1 (see formula (|11.7|) ). then 

Fm{A 2 ) = {fj^i + §w 2 ; s + si + s 2 = M, s 0l si,s 2 eZ- }. 

It is seen from here that the vertices 0, u\, w 2 of the fundamental domain F(A 2 ) belong to each 
grid Fm{A2). A direct computation shows that in the w-coordinates we have 

F 2 (A 2 ) = {(0,0),(l,0),(0,l),(i,0),(0,i),(i,i)}, 

F 3 (A 2 ) = {(0,0),(l,0),(0,l),(i,0),(0,i),(|,0),(0,|),(|,i),(i,f),(i,i)}. 

In F 2 (A2), only the point Q, ^) does not belong to a wall of the fundamental domain F(A 2 ). 
In ^3(^2); three points Q, |), (|, i), (|, i) do not belong to a wall of F{A2). 
The set K^^) consists of the points 

F 5 {A2) = {(5, 5), ( 5 , 5 ), (5, 5), (g, |), ( 5 , 5 ), (5, 5)} . 

Since the long root £ of C2 is representable in the form £ = 2ai + 02, where «i and a 2 are 
simple roots, that is, mi = 2,ni2 = 1, then 

F M {C 2 ) = {fr^i + §w 2 V ; so + 2si + s 2 = M, so, si, s 2 € Z^ } . 
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A direct computation shows that in the w v -coordinates we have 

F 2 (C 2 ) = {(0,0),(0,1),(±,0),(0,±)}, 

F 3 (C 2 ) = {(0,0), (0, 1), (1,0), (0, |), (0, |), (I, i)} , 

F-j (C 2 ) = {(7, 7), (7, 7), (7, 7), (7, 7), (7, 7), (7, 7)} • 

Since the long root £ of G 2 is representable in the form £ = 2ct\ + 3a2, where a\ and a 2 are 
simple roots, that is, mi = 2, 777-2 = 3, then 

F M {G 2 ) = { + §w 2 v ; s + 2 81 + 3s 2 = M, s , s u s 2 G lP"} . 

A computation shows that in the w v -coordinates we have 



F 2 (G 2 ) = {(0,0),(1,0)}, 

F 3 (G 2 ) = {(0,0),(0,±),(±,0)}, 

F 4 (G 2 )=F 2 (G 2 )u{(i,0),(0,i)}, 

F 5 (G 2 ) = {(0,0),(0,I),(I,0),(i,I),(|,0)}, 

F 8 (G2) = F 4 (G2)u{(I,0),(0,|),(i,i),(I,i)}, 

14' 14)' (l4' 7)' (l4' 14)' (7' 14)' (7' 7)' 
(^4"' 14)' (li' 7)' Vfi 14 )' (li' Ti)} • 



11.4 Expanding in antisymmetric orbit functions through finite sets 

Let us give an analogue of the finite Fourier transform when instead of exponential functions we 

use antisymmetric orbit functions. This analogue is not so simple as finite Fourier transform. 

For this reason, we consider some weak form of the transform. In fact, we consider this weak 

form in order to be able to recover (at least approximately) the decomposition f(x) = ^ a\ip\(x) 

x 

by values of f{x) on a finite set of points. 

Considering the finite Fourier transform in Section 111.11 we have restricted the exponential 
function to a discrete set. Similarly, in order to determine a finite transform, based on anti- 
symmetric orbit functions, we have to restrict orbit functions (f\(x) to appropriate finite sets of 
values of x. Candidates for such finite sets are sets T m . However, antisymmetric orbit functions 
<P\(x) with integral A are invariant with respect to the affine Weyl group W &s . For this reason, 
we consider orbit functions ^a(^) on grids Fm- 

On the other side, we also have to choose a finite number of antisymmetric orbit functions, 
that is, a finite number of integral strictly dominant elements A. The best choice is when 
a number of orbit functions coincides with \Fm\- These antisymmetric orbit functions must be 
selected in such a way that the matrix 

(^^))x^ jeFM (H-8) 

(where £1 is our finite set of strictly dominant elements A) is not singular. In order to have non- 
singularity of this matrix some conditions must be satisfied. In general, they are not known. 
For this reason, we consider some, more weak, form of the transform (when \Q\ > |-Pm|) and 
then explain how the set \Q\ of A G can be chosen in such a way that |f2| = \Fm\- 

Let O(A) and 0(/i) be two different VK-orbits for integral strictly dominant elements A and \i. 
We say that the group T m separates O(X) and 0(/i) if for any two elements Ai G O(A) and 
fii G 0(/i) there exists an element x £ T m such that exp(27ri(Ai, x)) ^ exp(27ri( / ui, x)) (we use 
here orbits, not signed orbits, since signs of points are not important for this reasoning). Note 
that A may coincide with \x. 
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Let /i and /2 be two functions on E n which are finite linear combinations of orbit functions. 
We introduce a T m -scalar product for them by the formula 

</i,/ 2 >T m = £ fi(x)Mx). 

xeT m 

Then the following proposition is true (see [21] and |31j): 

Proposition 17. IfT m separates the orbits O(X) and O(fi), X,fi G Pf } then 

(<Px,^)T m =m n \W\5 Xfl . (11.9) 
Proof. We have 

{tpXiP^Tm = £ ^2 (detw)(detw')ex.p(2'Ki(w\-w'n,x)) 



(detuiu/) exp(27ri(wA 



w'n,x)) . 



Since T m separates O(X) and O(fi), then none of the differences wX — w'n in the last sum vanishes 
on T m . Since T m is a group, one has 

^ exp(27ri(wA - w'fj,,x)) = m n 5 wX>w >^. 

Therefore, {^px,^ffi)T m = m n |W| Proposition is proved. ■ 

Let / be an anti-invariant (with respect to VK aff ) function on E n which is a finite linear 
combination of antisymmetric orbit functions: 

f(x)= £ a Xj cp Xj (x). (11.10) 

Our aim is to determine f(x) by its values on a finite subset of E n , namely, on T m . 

We suppose that T m separate orbits O(Xj) with Xj from the right hand side of (jll.lOp . Then 
taking the T m -scalar product of both sides of (jll.lOj) with ip Xj and using the relation (|11.9ft we 
obtain 

a Xj ={m n \W\r l {f^ X] ) Tm . 

Let now s^\ s^ 2 \ . . . , s^ h ' be all elements of F n ^<2 V , which do not lie on some wall of Weyl 
chambers. Then 



a X] =m- n \W\- 1 /W^W=m n E/( s(,) )^( s(!) )' ( 1L11 ) 
zeT m 1=1 

Thus, the finite number of values /(sW), i = 1, 2, . . . , h, of the function f(x) determines the 
coefficients a x . and, therefore, determines the function f{x) on the whole space E\. 

This means that we can reconstruct a W^-anti- invariant function f(x) on the whole space E n 
by its values on the finite set Fm under an appropriate value of M. Namely, we have to 
expand this function, taken on Fm, into the series (jll.lOp by means of the coefficients a Xj , 
determined by formula (jll.lip . and then to continue analytically the expansion (jll.lOp to the 
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whole fundamental domain F (and, therefore, to the whole space E n ), that is, to consider the 
decomposition (|11.1U|) for all x G E n . 

We have assumed that the function f{x) is a finite linear combination of orbit functions. 
If f{x) expands into infinite sum of orbit functions, then for applying the above procedure we 
have to approximate the function f{x) by taking a finite number of terms in this infinite sum 
and then apply the procedure. That is, in this case we obtain an approximate expression of the 
function f(x) by using a finite number of its values. 

At last, we explain how to choose a set £1 in formula (lll.8|) . The set Fm consists of the 
points (jll.6p . This set determines the set of points 

A = sicui + S2UJ2 H 1- s n uj n , 

where s±, S2, ■ ■ ■ , s n run over the same values as for the set Fm- The subset of this set, consisting 
of strictly dominant elements, can be taken as the set (see |31j). 

11.5 Antisymmetric and symmetric multivariate discrete Fourier transforms 

The discrete Fourier transform of Subsection 111.11 can be generalized to the n-dimensional case 
in a symmetric or antisymmetric form without using the results of Subsection 111.41 
We take the discrete exponential function (jll.ip . 

e m (s) := N^ 2 exp(2TTims), s G F N = {-fr, f, . . . , 1} , m G Z-°, 

and make a multivariate discrete exponential function taking a product of n copies of func- 
tions (fTLTJ) : 

EXP m (s) := e mi (si)e m2 (s 2 ) ■ ■ • e m „(s n ) (11.12) 

= N~ n / 2 exp(27rimisi) exp(27rim2S2) ■ ■ ■ exp(2-7rim n s n ), sj G Fn, rrii G Z-°, 

where s = (s\, S2, ■ ■ ■ , s n ) and m = (mi,m 2 , ■ ■ ■ , rn n ). Now we take these multivariate functions 
for integers mj such that mi > rri2 > ■ ■ ■ > m n > and make an antisymmetrization. As 
a result, we obtain a finite version of the antisymmetric orbit function (|4.14|) : 

e m (s) := |5 n r 1 / 2 det(e mi ( Sj ))^ =1 , (11.13) 

where \S n \ is the order of the symmetric group S n . 

The n-tuples s in (j 1 1 .131) runs over F^ = F/v x • • • x Fn {n times) . We denote by F^ the 
subset of F^ consisting of s G FJ^ such that 

si > s 2 > ■ ■ ■ > s n . 

Note that acting by the permutations w G S n upon FVi we obtain the whole set FZ without 
those points which are invariant under some nontrivial permutation w G S n . Clearly, the 
function (|11.13p vanishes on the last points. 

Since the discrete exponential functions e m (s) satisfy the equality e m (s) = e m+ ]y(s), we do not 
need to consider them for all values m G Z— . It is enough to consider them for m G {1, 2, . . . , iV}. 
By D~x we denote the set of integer n-tuples m = (mi, m 2 , • • • , m n ) such that 

N > mi > 771,2 > • • • > 777 n > 0. 

We need a scalar product in the space of linear combinations of the functions (| 1 1 . 12|) . It can 
be given by the formula 

n n 

(EXP m (s),EXFV(s)) = ]\{e mi {si),e mli {si)) := JJ ^ e mM)ertfii) = 6 mm', (H-14) 

i=l i=l Si£F?f 

mtX G {1,2,..., N}, 
where we used the relation (jll.2p . 
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Proposition 18. For m, m' £ the discrete functions (| 1 1 . 13|) satisfy the orthogonality rela- 
tion 

(e m (s),e m /(s)) = \S n \ ^2 e m(s)e m /(s) =6 mm >, (11.15) 
where the scalar product is determined by formula (|11.14p . 

Proof. Since m\ > mi > • • • > m n > 0, then due to the definition of the scalar product we 
have 

(e m (s),e m /(s)) = ^2 e m (s)e m /(s) 

seF™ 

n 

= \Sn\~ 1 £ II E e mw(i) (si)e m ^(si) = 5 mm/ , (11.16) 

where ("Wi^m,^)) • • ■ j m w(n)) is obtained from (mi,m2, . . . , m n ) by action by the permutation 
w £ S n . Since functions e m (s) are antisymmetric with respect to S n , then 

XI e m(s)e m /(s) = \S n \ ^2 e m (s)e m /(s), 

where we have taken into account that e m (s) vanishes on those s € Ffy for which there exists 
w S S n , w 7^ 1, such that ws = s. This proves the proposition. ■ 

Let / be a function on FJtj (or an antisymmetric function on -Fjy). Then it can be expanded 
in the functions f)l 1 . 13j) as 

f(s)= J2 «me m (s). (11.17) 

meD+ 

The coefficients (i ui cirG determined by the formula 

a m = \S n \ Yl /( s )^(s)- (11-18) 

The expansions (|11.17p and f)l 1 . 18|) follow from the facts that numbers of elements in D~t, and 
in Fjlj are the same, and that the matrix 

(e m (s)) m6jD + jSe ^„ 

is unitary. We call expansions (|11.17p and f) 1 1 . 1 8 1) antisymmetric multivariate discrete Fourier 
transforms. 

Let us also give a symmetric multivariate discrete Fourier transforms. For this we take the 
multivariate exponential functions fj 1 1 . 12|) for integers mi such that 

N > m\ > 7712 > • • • > TTln > 1 

and make a symmetrization. We obtain a finite version of the symmetric orbit function (6.11) 
in [1] for the case A n : 

n 

E m (s) := \S n \- l / 2 det + {e mi {s 3 ))t j=1 := \Sn\~ 1 ' 2 £ U e ^ s ^ (H-19) 
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The n-tuples s in (|11.19p run over Ffi = x • • • x Fjy (n times). We denote by Ffi the 
subset of F^j consisting of s G FJ$ such that 

si > s 2 > ■ ■ ■ > s n . 

Note that acting by the permutations w G S n upon F^ we obtain the whole set Ffi, where each 
point, having some coordinates m; coinciding, is repeated several times. Namely, a point s is 
contained \S S \ times in {wF^;w G S n }, where S s is the subgroup of S n consisting of elements 
w G S n such that ws = s. The number \S S \ is called a multiplicity of the point s in the set 
{wFjfr;weS n }. 

By we denote the set of integer ra-tuples m = (m 1 ,m 2 , . . . , m n ) such that 
N > mi > m 2 > • • • > m n > 1- 

Proposition 19. For m, m' G Z)^- the discrete functions (| 1 1 . 19|) satisfy the orthogonality rela- 
tion 

{E m {s),E m ,( S )} = \S n \ Y \S s \- 1 E m (s)E~Asj=\S m \5 mm ,. (11.20) 

Proof. This proposition is proved in the same way as Proposition 1181 but we have to take into 
account a difference between Ffy and Fife. Due to the definition of the scalar product we have 



<£m(s),£m'(s)>= Y £m(s)£m'(s) 



— I'S'nl 1 |5'm| II em Mi)^ Si ^ e m' w{i) ( s i) 

W&Sn 1=1 Si£F N 

= I Sm 1 <5 mm / , 

where (m w n\, m w (2), • • • , m w (n)) is obtained from (mi, m-2, . . . , m n ) by action by the permutation 
w £ S n . Here we have taken into account that additional summands appear (with respect to 
(|11.16p ) because some summands on the right hand side of (|11.19p may coincide. 
Since functions E m (s) are symmetric with respect to S n , then 

Y E m (s)E^Asj = \S n \ Y |5 s r 1 F m (s)F^), 

where we have taken into account that under action by S n upon FVj a point s appears |5 S | times 
in FJtj. This proves the proposition. ■ 

Let / be a function on FJ^ (or a symmetric function on Ffy). Then it can be expanded in 
functions (|11.19|) as 

/(S)= a m£m(s). (11.21) 

meD+ 

The coefficients a m are determined by the formula 



o r 



\s n \\s m \~ i y \s s r 1 f(s)E m ( S ). (n.22) 



The expansions (jll.2ip and (|11.22p follow from the facts that numbers of elements in 
and in F^ are the same and from the orhogonality relation (jll.2U|) . We call expansions (|11.21|) 
and (|11.22p symmetric multivariate discrete Fourier transforms. 
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11.6 Discrete sine and cosine transforms 

The grid Fm for A\ is of the form 

F M (A 1 ) = {0,±,±,...,M KT 1 A }. (11.23) 

The points and 1 belong to the boundary of the fundamental domain F(A\) of VF aff (^4i). 
Therefore, antisymmetric orbit functions of A\ vanish on these points and 

F m(Ai) = {w,w,---, M m 1 } (M-l points) 

(see Subsection lll.2p . Since the antisymmetric orbit functions for A± are of the form <p\(x) = 
2isin(7rm0) (see Example of Subsection 14. ip . these functions on the grid Fm are given by 

fm{ s ) = 2isin(7rms), s G Fm, m G Z-. (11.24) 

Since tp m (s) = <p m +M(s), we consider these discrete functions only for m G Dm '■= {1,2,... , 
M — 1}. The orthogonality relation for these functions is of the form 

( ( Pm,Pm') = ^ <Pm(s)<p m '(s) = 2M5 mm >, to, to' G Dm 
(see [22]). They determine the following expansion of functions, given on the grid F M : 

M-l 

f( s ) = ^ a m^m{s), (11.25) 
m=l 

where the coefficients a m are given by 

a ™ = 2F S ( 1L26 ) 

s€F M 

Formulas (|11.25j) and (jll.26p determine the discrete sine transform. 

The symmetric orbit functions for A± are of the form (j)\(x) = 2cos(7tto#). Then these 
functions on the grid Fm{A{) are 

(j) m (s) = 2cos(vrms), s G Fm, m G {0, 1, 2, . . . , M}. (11.27) 

The scalar product of these functions is given by 

(4>m,4>m'}= ^ c s<Pm{s)4> m >(s) = r m MS mm >, (11.28) 

seF M 

where r m = 4 for m = 0, M and r m = 2 otherwise, c s = 1/2 for s = 0, 1 and c s = 1 otherwise. 
The functions <^ m , given by (jll.27p . determine an expansion of functions on the grid Fm as 

M 

f(s) = KK(s), s G F M , (11.29) 

m=0 

where the coefficients b m are given by 

bm = r~ 1 Y c s f{s)4> m {s). (11.30) 
seF M 

Formulas (jll.29p and (|11.30p determine the discrete cosine transform. 
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11.7 Antisymmetric multivariate discrete sine transforms 

The discrete sine and cosine transforms of the previous subsection can be generalized to the 
ra-dimensional case in symmetric or antisymmetric form. In fact, these generalizations are finite 
antisymmetric orbit function transforms (jll.lOp and ([11. lip for multivariate sine transforms 
and finite symmetric orbit function transforms (9.11) and (9.12) in pQ for multivariate cosine 
transforms. However, we give in this subsection a derivation of these multivariate sine and cosine 
transforms, independent of the previous consideration. We need only the 1-dimensional discrete 
sine and cosine transforms of the previous subsection. 

We take the discrete sine function (jll.24p and make a multivariate discrete sine function by 
multiplying n copies of functions (I11.24P : 

SIN m (s) := (2i) n sin(7rmiSi) sin(7rm 2 S2) ■ ■ ■ sin(7rm n s n ), (11.31) 
Sj E F M = Fm(A{), rm E D M = {1,2, . . . , N - 1}, 

where s = (si, s 2 , ■ ■ ■ , s n ) and m = (mi,m2, ■ • ■ , m n ). Now we take these multivariate functions 
for integers rrii such that M > m\ > m 2 > • • • > m n > and make antisymmetrization. As a 
result, we obtain a finite version of the orbit function (|4.19p : 

V9 m (s) := (2i) n |5„r 1 / 2 det(sin7rm i s i )^ =1 , (11.32) 

where \S n \ is the order of the symmetric group \S n \. (We have here expressions sin7rmjSj, not 
sin 2irmiSj as in (|4.19p . Note that in (|4.17p rrii, i = 1, 2, . . . , n, run over integers and half-integers, 
whereas in (jll.32p m 8 run over integers. Thus, in fact we have replaced 2m j with half- integer 
values of rrii by m i with integer values of rrii.) 

The n-tuple s in (|11.32p runs over F^(Ai) n = F^(Ai) x • • • x F^(Ai) (n times). We denote 
by Fm the subset of F^(Ai) n consisting of s G F^(Ai) n such that 

Si > S 2 > ■ ■ ■ > S n - 

Note that here may take the values jj, jj, ■ ■ ■ ■> ^jt~- Acting by permutations w € S n upon F^ 
we obtain the whole set F^ I {A\) n without those points which are invariant under some nontrivial 
permutation w E S n . Clearly, the function (jll.321) vanishes on the last points. 
We denote by D~j^ the set of integer n-tuples m = (mi, m 2 , ■ ■ ■ , m n ) such that 

M > m\ > m 2 > ■ ■ ■ > m n > 0. 

We wish to have a scalar product of functions (|11.32j) . For this we define a scalar product of 
functions (|11.3ip as 

n 

(SIN m (s),SIN m ,(s)} = Y[{<p mi (8i),<p < (s i )), 

i=i 

where the scalar product ((p mi (si), <p m '. (sj)) is given in Subsection 111.61 Since functions v? m ( s ) 
are linear combinations of functions SIN m /(s), a scalar product for <p m (s) is also defined. 

Proposition 20. For m, m' G D^, the discrete functions (11 1 .321) satisfy the orthogonality 
relation 

(Pm(s),^m'(s)) := ^2 <^m(s)^m'(s) = \S n \ ^ ip m (s) Cf m > (s) = {2M) n 5 mm >. (11.33) 
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Proof. Since M > m\ > ?n 2 > • • • > m n > 0, then due to the orthogonality relation for the 
sine functions 2isin(7rms) (see the previous subsection) we have 

n M-l 

^2 ^m(s)9?m'( s ) = ^ISVJ" 1 ^ ] [ Yj sin (^ m ^) Si ) Sin (™Mi)' Si ) = ( 2M ) n ^mm', 

s6F^(Ai)« w£S n i=lSi=l 

where (m w ^, m w ^), . . . , rn w ( n )) is obtained from (mi, to 2 , . . . , m n ) by action by the permutation 
w G S n . Since functions V3 m (s) are antisymmetric with respect to S n , we have 

X! ^m(s)</?m'( s ) = |^n| 2j ( s ) ¥W (s) . 

This proves the proposition. ■ 

Let / be a function on (or an antisymmetric function on F^(Ai) n ). Then it can be 
expanded in functions (|11.32|) as 

/(S)= Yl a m^m(s), (11.34) 
meD+ 

where the coefficients a m are determined by the formula 

a m = {2M)- n \S n \ Y, /( s Ws)- (U-35) 

A validity of the expansions (|11.34p and (|11.35p follows from the facts that numbers of 
elements in and in FJ^ are the same and from the orthogonality relation (jll.33f) . 

11.8 Symmetric multivariate discrete cosine transforms 

We take the discrete cosine functions (|11.27j) and make multivariate discrete cosine functions by 
multiplying n copies of these functions: 

COS m (s) := ^ mi (si)0 m2 (s 2 ) • • • <pm n (s n ) = 2 n cos(7rmiSi) cos(7rm 2 s 2 ) • • • cos(7rm n s n ), 

Sj G Fm = Fm(A\), m t G {0,1,2,..., M}. (11.36) 

We take these functions for integers m,j such that M > m\ > mi > ■ ■ ■ > m n > and make 
a symmetrization. As a result, we obtain a finite version of the orbit function (|4.24|) : 

</>m(s) : = 2 n |5 n |" 1/2 Y cos7rm w (i)Si • cos7rm u ,( 2 )S 2 • • • cos 7r?n w ( n) s n . (11.37) 

(We have here expressions cosnmiSj, not cos2irmiSj as in (|4.24p .) 

The ?i-tuple s in (111.371) runs over Ffa = F M {Ai) n . We denote by FJfr the subset of Ffo 
consisting of s G Fjfc such that 

si > s 2 > • • • > s n . 

Note that here may take the values 0, . . . , , 1. Acting by permutations w £ S n 

upon we obtain the whole set Ffy, where points, invariant under some nontrivial permutation 
w G S n , are repeated several times. It is easy to see that a point so G Ffa is repeated | -S'so I times 
in the set {wF^; w G S n }, where \S S(I \ is an order of the subgroup 5 So C S n , whose elements 
leaves Sq invariant. 
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We denote by the set of integer n-tuples m = (roi,m2, ■ ■ ■ , m n ) such that 
M > mi > m 2 > • • • > Tn n > 0. 

A scalar product of functions (|11.36|) is determined by 

n 

<COS m (s),COS m ,(s)> = n^(si),^(si)), 

i=i 

where the scalar product {4> mi (si), 4> m i, (si)) is given by (jll.28p . Since functions m (s) are linear 
combinations of functions COS m '(s), then a scalar product for 4> m (s) is also defined. 

Proposition 21. For m, m' £ ^ti> ^he discrete functions (jll.37p satisfy the orthogonality 
relation 

(0m(s),0 m '(s)) = ^ C s m (s)^) m /(s) = |5 n | ^ |5 S | ~ 1 C s (^) m (s)0 m / (s) 

= M n r m |5 m |5 mmS (11-38) 

w/iere c s = c Sl c S2 ■ ■ ■ c Sn , r s = r mi r m2 ■ ■ ■ r mn , and c Si and r m% are such as in (|11.28p . 

Proof. Due to the orthogonality relation for the cosine functions <j)m(s) = 2cos(-7rms) (see 
formula (|11.28p ) we have 

n M 

^ Cs0 m (s)0 m /(s) = 4 n |5 n | _1 |5 m | ^2 ] [ X! Cs i COs(TTm w ^ySi) cos(TTm w(i) Sj) 
s€F™ w£S n i=l s,=0 

= \S m \M n r m 5 mm ,, (11.39) 



where (rn w m, m w ( 2 ), . . . , rn w ^) is obtained from (mi,m2, . . . , m n ) by action by the permutation 
w £ S n . Since functions </> m (s) are symmetric with respect to S n , we have 



C s 0m(s)</>m'(s) = |5 S | 1 C s (?!> m (s)0 m /(s). 

s€F M (^i) n s€F« 

This proves the proposition. ■ 

Let / be a function on FJfo (or an antisymmetric function on FV^). Then it can be expanded 
in functions (|11.37p as 

/(B) = Yl a m0m(s), (11.40) 

where the coefficients a m are determined by the formula 
a m = M- n |5 m |- 1 r m 1 (/(s),0 m (s)) 

= M" n |5 m |- 1 r m 1 |5n| |5»rW(s)Mi)- (11-41) 



A validity of the expansions (lll.40p and ()11.41|) follows from the fact that numbers of elements 
in and FJ^ are the same and from the orthogonality relation (jll.38p . 
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11.9 Other discrete cosine transforms 

Along with the discrete cosine transform of Subsection 111.61 there are other discrete transforms 
with the discrete cosine function as a kernel. In [M] the discrete cosine transforms are called as 
DCT-1, DCT-2, DCT-3, DCT-4. The transform DCT-1 is in fact the transform, considered in 
Subsection 111.61 Let us describe all these transforms (including the transform DCT-1). They 
are determined by a positive integer N. 

DCT-1. This transform is given by the kernel 

fj, r (k) =cos^ (11.42) 

(we preserve the notation used in the literature on signal processing), where 

k,r £ {0,1,2, ... ,7V}. (11.43) 
The orthogonality relation for these discrete functions is given by 

E-Krk irr'k N 
c k cos — cos -jj- = h r —d rr >, (11.44) 

k=0 

where = 1/2 for k = 0, N and c& = 1 otherwise, h r = 2 for r = 0, N and h r = 1 otherwise. 
Thus, these functions give the expansion 

,/m £ ^k 2 ^ ™k , , 

J W = Z^ arCOS_ /^' where a r = j— ^ c k f{k) cos — . (11.45) 

DCT-2. This transform is given by the kernel 
7r( t -\- — ) k 

u) r {k) = cos V 2J , (11.46) 
where 

k,r £ {0,1,2,... , N- 1}. 
The orthogonality relation for these discrete functions is given by 

\^ *(r+h) k <r' + \)k N . . 

> cu cos — cos — = — o rr i, 11.47 

^ N N 2 

fc=0 

where c k = 1/2 for k = and c k = 1 otherwise. 
These functions determine the expansion 

N-i 2 N ~ 1 

f( k ) = ^2 a r u r(k), where a r = — ^ c k f(k)u r (k). (11.48) 

r=0 fc=0 

DCT-3. This transform is determined by the kernel 
■jtr( fa 4- 1^ 

<r r (fc)=C0B ^ , (11-49) 
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where k and r run over the values {0,1,2,... ,N — 1}. The orthogonality relation for these 
discrete functions is given by the formula 

V? *r(fc+j) *r\k + \) N 

2^ cos — cos — = hr—5 rr >, (11.50) 

k=0 

where hk = 2 for k = and hk = 1 otherwise. 
These functions give the expansion 

fi k ) = ^2 ° r cos — Sv~~^' where flr = Or ^ ^ cos ~~ OT 2 ^ ( 1:L - 51 ) 

r=0 r fc=0 

DCT-4. This transform is given by the kernel 

Tr (k) = cos -i ^ , (11.52) 

where k and r run over the values {0,1,2,... ,N— 1}. The orthogonality relation for these 
discrete functions is given by 

™ vr(r + l)(fe+i) K {r i + \){k+\) N 

> cos — cos — = — o rr i. 11.53) 

^ N N 2 v ' 

k=0 

These functions determine the expansion 

/(fe)=£\ r cos^±M±il, (11.54) 

r=0 



where 



ar = N^ f{k) C ° S N • 

fc=0 

Note that there also exist four discrete sine transforms, corresponding to the above discrete 
cosine transforms. They are obtained from the cosine transforms by replacing in (lll.45p . (I11.48j) . 
(|11.5ip and (|11.54p cosines discrete functions by sine discrete functions (see [12] and [35]). 



11.10 Other antisymmetric multivariate discrete cosine transforms 

Each of the discrete cosine transforms DCT-1, DCT-2, DCT-3, DCT-4 generates the corres- 
ponding antisymmetric multivariate discrete cosine transforms. We call these transforms as 
AMDCT-1, AMDCT-2, AMDCT-3 and AMDCT-4. Let us give these transforms without proof. 
Their proofs are the same as in the case of antisymmetric multivariate discrete cosine transforms 
of Subsection II 1.71 Below we use the notation D^f for the subset of the set D 1 ^ = Dn x Dn x 
• • • x Dn (n times) with = {0, 1, 2, ... , N} consisting of points r = (n, r2, ■ ■ ■ , r n ), rj G Dn, 
such that 



N > r% > ri > ■ ■ ■ > r n > 0. 



AMDCT-1. This transform is given by the kernel 

M r (k) = \S n \-^ 2 det {ii n {kj)) n ij= i , (H-55) 
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where fi r (k) = cos and k = (fci, & 2 , . . . , ki G {0, 1, 2, ... , TV}. The orthogonality relation 
for these kernels is 

(M r (k),M r /(k)) = |5 n | ^ c k M r (k)M r /(k) = h r l—j <W, (11.56) 



where 

ck = cic 2 • • -c n , hk = hih 2 ■ ■ ■ h n , 

and Cj and /ij are such as in formula (|1 1.44|) . 
This transform is given by the formula 



/(k)= Yl M4(k), where a r = h^Sn] E c k /(k)M r (k). (11.57) 



The Planchcrcl formula for this transform is 



\S n \ E ^l/(k)| 2 =(y)" E h 



|2 

r iCl r . 



AMDCT-2. We use the subset L>^'_i of the set D%_ x with D N ^ = {0, 1, 2, . . . , N - 1} 
consisting of points r = (ri, r 2) . . . , r n ), r» S Dn-i, such that 

AT — 1 > ri > r 2 > • • • > r n > 0. 

This transform is given by the kernel 

n r (k) = IS^-^det (u n (kj))Z j=1 , (11-58) 

7r(r+ — )k 

where uj r {k) = cos — — and k = [ki, fc 2 , ■ ■ ■ , k n ), ki G {0, 1, 2, . . . , N — 1}. The orthogonality 
relation for these kernels is 

(n r (k),n r ,(k)) = |5 n | E c k ft r (k)tt r ,(k) = f <*„,, (11.59) 

where c k = cic 2 • • • c n and Cj are such as in formula (jll.47p . 
This transform is given by the formula 



/(k) = E Or^r(k), where a T = \S n \ (— J E c k/(k)f2 r (k). 



(11.60) 



The Plancherel formula for this transform is of the form 

\Sn\ E ^l/(k)| 2 =(f) n E l«r! 2 - 

AMDCT-3. This transform is given by the kernel 

S r (k) = |5 n |- 1 / 2 detK(%))~ , r€D n N Z v % € D N - X , (11.61) 
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where a r (k) = cos — N 2 . The orthogonality relation for these kernels is 

(S P (k),S P /(k)) = |5„| Y S r (k)S r ,(k) = ^ r f-J (11.62) 

where /i r = h\h,2 ■ ■ ■ h n and hj are such as in formula (|11.50p . 
This transform is given by the formula 

/(k)= Y «rS r (k), where a r = h- 1 ^ Y /(k)E r (k). (11.63) 

The Plancherel formula is of the form 

AMDCT-4. This transform is given by the kernel 

r P (k) = |5 n |- 1 / 2 det(r ri (*! i ))? J=1I r€D%Z v kj € D N - lt (11.64) 

where T r (k) = cos 7T ^ k+ ^ r+ ^ m The orthogonality relation for these kernels is 



\Sn\ E l/( k )! 2 =(f) n E * 



i2 

r ifl r . 



f N \ n 

(T T (k),T r ,(k)) = \S n \ Y, T r (k)T r ,(k)= (- ) <5 rr ,. (11.65) 



This transform is given by the formula 



/(k)= £ a r T r (k), where a r = |5 n | (J^j Y /( k ) T r( k )- ( 1L66 ) 



The Plancherel formula for this transform is 

t)" 

ten"- x 7 r pD n '~ 



IS..I E l/«l 2 =(y)" E 



|a r | 2 . 



11.11 Other symmetric multivariate discrete cosine transforms 

To each of the discrete cosine transforms DCT-1, DCT-2, DCT-3, DCT-4 there corresponds 
a symmetric multivariate discrete cosine transform. We denote the corresponding transforms as 
SMDCT-1, SMDCT-2, SMDCT-3, SMDCT-4. Below we give these transforms without proof 
(proofs are the same as in the case of symmetric multivariate discrete cosine transforms of 
Subsection 111. 8D . We fix a positive integer N and use the notation for the subset of 

the set .D^r = Djy x x • • ■ x Dn (n times) with = {0,1,2,..., N} consisting of points 
r = (n, 7*2, . . . , r n ), ri G Z-° such that 

N > r\ > r 2 > • • • > r n > 0. 

The set D^J + is an extension of the set from the previous subsection by adding points 

which are invariant with respect of some elements of the permutation group S n . 
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The set D 7 ^ is obtained by action by elements of the group S n upon D 7 ^ , that is, coin- 
cides with the set {wD^ + ; w E S n }. However, in E S n }, some points occur several 
times. Namely, a point ko E D^J + occurs |<Sk | times in the set {wD^J + ;w E S n }, where |<Sk | i s 
an order of the subgroup Sk C S n consisting of elements w E S n leaving ko invariant. 

SMDCT-1. This transform is given by the kernel 

M r (k) = \S n \~ 1 / 2 ]T f i rwW (k 1 ) f i rw(2) (k 2 )--- f i rw(n) (k n ), (11.67) 
wes„ 

where, as before, fj, r (k) = cos z ^ is the discrete cosine function from Subsection 111.91 k = 
(hi, k 2 , . . . , k n ), ki E {0, 1, 2, ... , N}, and the set (w(l),w(2), . . . , w(n)) is obtained from the set 
(1,2, ... ,n) by applying the permutation w E S n . The orthogonality relation for these kernels 
is 

(M r (k),M r ,(k)) = |5 n | |5 k |- 1 c k M r (k)M r ,(k) = / lr f -\ \S r \8 rr> , (11.68) 

where S r is the subgroup of S n consisting of elements leaving r invariant, 
Ck = cic 2 •■•Cm /ik = hih 2 ■ ■ ■ h n , 

and Q and hj are such as in formula (111. 44ft . 
This transform is given by the formula 

/(k)= Yl a r M r (k), (11.69) 

where 

2\ n 



o r = / l - 1 |5 r |- 1 |5 n | (^) Y l^kr 1 c k /(k)A/ r (k). 



The Plancherel formula for this transform is 

\Sn\ £ |5k|- 1 C k |/(k)| 2 = ]T MS 



i2 

r 1 1 fl r | . 



This transform is in fact a variation of the symmetric multivariate discrete cosine transforms 
from Subsection 111.81 

SMDCT-2. This transform is given by the kernel 

{l r (k) = \Sn\- 1 ' 2 ]T C0 rwm (k 1 )u rw{2) (k 2 )---U rw(n) (k n ), (11.70) 

wes n 

r E D%± v Tj E Djv-l = {0, 1, 2, . . . , N - 1}, 

where oj r (k) = cos — ^ — and k = (k\, k 2 , ■ ■ ■ , fc n )> fej G {0, 1, 2, . . . , iV — 1}. The orthogonality 
relation for these kernels is 

/ AT\ n 

(n r (k),n r /(k)) = |5 , „| \Sk\- 1 c k h r (k)fi r ,(k) = (-) \s r \s rr ,, (11.71) 
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where D^_^ is the set D^J + with N replaced by N — 1, c k = c\c 2 ■ ■ ■ c n and cj are such as 
in ([TOTD . 

This transform is given by the formula 

/(k)= Yl «A(k), (11.72) 



where 

a r = |S„||5 r |- 1 ^ £ \S k \- l c k f(m r (k). 

The Plancherel formula for this transform is of the form 

\S n \ J2 l*l _1 ck|/(k)| a = (yV £ l^||a r | 2 . 

SMDCT-3. This transform is given by the kernel 

£ r (k) = \S n \- 1 / 2 £ °r wW (k 1 )a rw(2) (k 2 )---a rw(n) (k n ), (11.73) 

r G -C^i, G Av-i, 
where cr r (A;) = cos ^~ 3 ^ . The orthogonality relation for these kernels is 

(E r (k),E r ,(k)) = |5 n | £ |5 k r 1 E r (k)E r /(k) = h r ( — \ \S T \5„,, (11.74) 
ke^+ V " J 

where h T = hih 2 • • • h n and hi are such as in formula (|11.50p . 
This transform is given by the formula 

/(k)= £ «rS r (k), (11.75) 



where 

a r = hi 



l \SA^\S n \(^) n £ |5 k |-V(k)S r (k). 



The Plancherel formula is of the form 

\s n \ £ |s k rV(k)| 2 = (y) £ MS 



i2 

r I l^r 



SMDCT-4. This transform is given by the kernel 

f r (k) = \S n \~ 1 / 2 £ rr wm (ki)r rw{2) (k 2 )...T rw{n) (k n ), (11.76) 



r G D^!, rj G -Div-i, 
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where T r (k) = cos 7T ^ k+ ^ r+ ^\ The orthogonality relation for these kernels is 

(T T (k),T r ,(k)) = \S n \ |5 , k r 1 T r (k)T r ,(k)= f- J |S r |<5 rr ,. (11.77) 

This transform is given by the formula 

/(k)= a ^(k), (11.78) 



where 



\ n 



The Plancherel formula for this transform is 

|2 

I 1 1 



12 Solutions of the Laplace equation 
on n-dimensional simplexes 

We have seen in pQ that symmetric orbit functions are solutions of the Neumann boundary 
value problem on n-dimensional simplexes. That is, they are solutions of the Laplace equation 
Af(x) = Xf(x) on the fundamental domain F of the corresponding affine Weyl group W 
satisfying the Neumann boundary condition 



d(j>x(x) 



dm 



= 0, AeP + , (12.1) 

dF 



where dF is the (n — l)-dimensional boundary of F and m is the normal to the boundary. In 
this section we show that antisymmetric orbit functions are solutions of the Laplace operator, 
which vanish on the boundary dF of the fundamental domain F. 



12.1 The case of n-dimensional simplexes related to A n , B n , C n and D n 

Let F be the fundamental domain of one of the affine Weyl groups W^(A n ), W aS (B n ), 
M^ aff (C n ), W^ aff (-D n ) (see Subsection 15.91 for an explicit form of these domains). We use orthogo- 
nal coordinates xi,X2, ■ ■ ■ ,x n +\ on F in the case of W a ' s {A n ) and the orthogonal coordinates 
x±,X2, ■ ■ ■ ,x n in other cases (see Section [3|). Thus the fundamental domain F for W (A n ) is 
placed in the hyperplane x\ + X2 + • • • + x n+ \ = 0. 

The Laplace operator on F in the orthogonal coordinates has the form 

dx\ dx\ dx% ' 

where r = n + 1 for A n and r = n for B n , C n and D n . Let us consider the case B n . We take 
a summand from the expression (|4.16p for the antisymmetric orbit function <f\(x) of B n and act 
upon it by the operator A. We get 

^ e 27ri((io(eA))ia;iH h(w(e\)) n x n ) 
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= -4vr 2 [(eim 1 ) 2 + • • • + ^ nmn f] e ^((H^))ixi+---+(^))nx n ) 
= -4m 2 {m\ + ■■■ + m 2) e 2-i((»( £ A)) 1 x 1 +...+(«,( £ A))„x n ) 

= -4vr 2 (A, A) e 27Ti ((w(e\))ix 1 +-+{w(e\)) n x n ) ^ 

where A = (mi,m2,...,m fI ) is the weight, determining ip\{x), in the orthogonal coordinates 
and w S S n . Since this action of A does not depend on a summand from (|4.16|) . we have 

Aip x (x) = -4vr 2 (A,A)v9 A (x). (12.2) 

For A n , C n and D n this formula also holds and the corresponding proofs are the same. Remark 
that in the case A n the scalar product (A, A) is equal to 

(A, A) = m\ + m\ H hm 2 +1 . 

The formula ()12.2j) can be generalized in the following way. Let &k(yi,y2, ■ ■ ■ , Ur) be the fc-th 
elementary symmetric polynomial of degree k of the variables y±, yi, ■ ■ ■ , y r , that is, 

<Tk(yi,V2, • • • ,y r ) = Yl Vkiyk 2 ■ ■ ■ Vk r - 

l<ki<k2<-<k r <n 



Then 



q2 q2 q2 



) tp x (x) = (-4ir 2 ) k a k (ml,ml...,m^ x (x), k = 1, 2, . . . , r, (12.3) 



where, as before, r = n + 1 for ^4 n and r = n for S n , C n and -D n . r differential equations () 1 2 . 3 1) 
are algebraically independent. 

Thus, antisymmetric orbit functions are eigenfunctions of the operator a k ^J^r, J^r, • • • , -§^z^ , 
k = 1, 2, . . . , n, on the fundamental domain i 7 satisfying the boundary condition 



^ A 0r)=O, AeD + , (12.4) 
(see Subsection 15. 3p . 

12.2 The Laplace operator in u;-basis 

We may parametrize elements of F by coordinates in the w-basis: x = 9iU)\+- ■ -+#2^2- Denoting 
by dk partial derivative with respect to 9k, we have the Laplace operator A in the form 

1 - 

A = 2 E (<*3><Xj)~ lM iA 9 j> ( 12 - 5 ) 

i,i=i 

where (Mij) is the corresponding Cartan matrix. One can see that it is indeed the Laplace 
operator as follows. The matrix (Sij) = {{aj,dj)~ l Mij) is symmetric with respect to transpo- 
sition and its determinant is positive. Hence it can be diagonalized, so that A becomes a sum 
of second derivatives with no mixed derivative terms. 

12.3 Rank two and three special cases 

We write down explicit form of the Laplace operators in coordinates in the w-basis for ranks 2 
and 3. For rank two the operator A is of the form 

A 2 : (dl-d 1 d 2 + d1)ip = -^(a 2 + ab + b 2 )p, F = {0, wi, w 2 }, (12.6) 
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C 2 : {2d 2 -2d 1 d 2 + d 2 2 )p = -2ir 2 {a 2 + 4ab + 4b 2 )ip, F = {0, u u cj 2 }, (12.7) 
G 2 : (d 2 -3d 1 d 2 + 3d 2 2 )v = -^(3a 2 + 3ab + b 2 )ip, F = {0,^,uj 2 }. (12.8) 

Here, to simplify notation, (p stands for (p\(x), A = (a b) and x = (6\ 9 2 ). Although the 
same symbols are used for analogous objects in the three cases, their geometric meaning is very 
different. It is given by the appropriate Cartan matrix M from (|2.ip . In particular, the vertices 
of F form an equilateral triangle in the case of A 2 , for C 2 the triangle is half of a square, and it 
is a half of an equilateral triangle for G 2 . 

In the semisimple case A± x A\ one has M = 2 ( J ° ), therefore A = 2d 2 + 2d 2 , and ^px(x) is 
the product of two antisymmetric orbit functions, one from each A\. The fundamental domain 
is a square. 

There are three 3-dimensional cases to consider, namely A3, B 3 , and C3. In addition there 
are four cases involving non-simple groups of the same rank. For ^3, B 3 , and C3 the result can 
be represented by the formulas 

A 3 : A = df + d 2 + d 3 - d\d 2 - d 2 d 3 , 
B 3 : A = d 2 + d 2 + 2d 2 - d\d 2 - 2d 2 d 3 , 

C 3 : A = 2d\ + 2d\ + 2<9f - 2d\d 2 - 2d 2 d 3 . (12.9) 



12.4 Antisymmetric orbit functions as eigenfunctions of other operators 

Antisymmetric orbit functions are eigenfunctions of many other operators. We consider examples 
of such operators. 

We associate with each y £ E n the shift operator T y which acts on the exponential functions 

e 2TTi(\,x) ag 

y 27ri<A,x) = e 2iri(\,x+y) = e 2m(\,y) e 2m(\,x) _ 

The action of elements of the Weyl group W on functions, given on E n , is given as wf(x) = 
f{wx). For each y £ E n we define an operator acting on orbit functions by the formula 

D y = (det w)wT y . 

Then 

D y (px(x) = D y ^2 (detw)e 2ni{wX ' x) 

= ( det ^0 Y {detw)e 2ni{wX > y) e 27Ti{wX > w ' x) 

w'GW w£W 

= Y,( detw ) e2wl{wX ' y) Y (detw')e 27Ti{wX ' w ' x) 
= Y( detw ) e2nl{wX ' y) Y {detw')e 27Ti{w '~ lwX > x) 
= Y e 2 ^ wX ^ Y {dztw'- l w)e 2 ^ w '~ lwX ^ 

= E <?™ {wX ' v) vx{x) = <l>x{y)<px{x), 

that is, (px( x ) is an eigenfunction of the operator D y with eigenvalue <f>x{y)- 
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Now we consider the operator 



4/ = J2 wT i 



Then, conducting the same reasoning as above, we receive the relation 



Dyifx(x) = tp\{y)ip\{x) 



that is, (f\(x) is an eigenfunction of the operator D y with eigenvalue (f\(y)- 
It is proved in the same way that 

D y (j)\{x) = ip x (y)4>x(x), 

that is, the symmetric orbit function 4>\(x) is an eigenfunction of the operator D y with eigenvalue 



It is shown similarly that in the cases of A n , B n , C n , D n antisymmetric orbit functions <p\(x) 
are eigenfunctions of the operators 



where x±, x 2 , ■ ■ • , x r are orthogonal coordinates of the point x, r = n + 1 for A n and r = n for 
other cases. In fact, these operators are multiple to the Laplace operator A. 

It is easy to show that in the cases of A n , B n , C n , and also of D n with even n, antisymmetric 
orbit functions <p\(x) are solutions of the equations 



13 Symmetric and antisymmetric functions 

Symmetric (antisymmetric) orbit functions are symmetrized (antisymmetrized) versions of the 
exponential function, when symmetrization (antisymmetrization) is fulfilled by a Weyl group. 
Instead of the exponential function we can take any other set of functions, for example, a set 
of orthogonal polynomials or a countable set of functions. Then we obtain a corresponding set 
of orthogonal symmetric (antisymmetric) polynomials or a set of symmetric (antisymmetric) 
functions. Such sets of polynomials and functions are a subject of investigation in this section. 

13.1 Symmetrization and antisymmetrization 
by (anti) symmetric orbit functions 

Symmetric and antisymmetric orbit functions can be used for symmetrization and antisym- 
metrization of functions. Let u m (x), m = 0, 1, 2, . . . , be a set of continuous functions of one 
variables. We create functions of n variables 

Ui ll i 2> ... t i n (x 1 ,x 2 , ...,x n ) = u il (xi)u i2 (x 2 ) ■ ■ ■ u in (x n ), i k = 0,1,2, 

Then the functions 

Ui 1 ,i 2 ,...,i n (Xi,X 2 ,---,Xn) = / Ui li i 2t ,„ ! i n (xi,x 2 , ■ ■ ■ ,x n )4> x {x 1 ,x 2 , . . . ,x n ,)dx, (13.1) 
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where A = (Ai, A2, • • • , A n ), <fi\(x) is a symmetric orbit function, and dx is the Euclidean measure 
on E n (that is, dx = dx\ ■ ■ ■ dx n ), is symmetric with respect to the action of the Weyl group W. 
Indeed, for w E W we have 



Ui lt i 2! ...,i n (w\) = J U iu i 2t ...j n (xi, X2, ■ ■ ■ ,X n )(f> w \(xi,X2, ■ ■ ■ , X n )dx 

= J iHui2,...,i n (Fi,X2, ■ ■ ■ ,x n )<f)\(xi,X2, ■ ■ ■ ,x n )dx = Ui 1 ,i a ,...,i n (X). 
Similarly, the functions 

Ui 1 ,i 2 ,...,i n (\i,\2,---,K) = / Ui lii2t ... !in (xi,X2, ■ ■ ■ ,x n )ip\(xi,x 2 , ■ ■ ■ ,x n )dx, (13.2) 



F 

where (p\(x) is an antisymmetric orbit function, are antisymmetric with respect to the action of 
the Weyl group W. In particular, the functions Ui lj i 2j ... : i n (X) vanish on Weyl chambers. 

Formulas (|13.ip and f)13.2|) are used for obtaining symmetric and antisymmetric functions or 
polynomials. 

13.2 Eigenfunctions of (anti) symmetric orbit function transform for W(A n ) 

Let H n (x), n = 0, 1, 2, ... , be the well-known Hermite polynomials. They are defined by the 
formula 

[™/ 2 ] / -\\m(r)^\n-2m 

H n (x)=nl^ { ] ( } 



m\(n — 2m)! 

m=o v ' 



where [n/2] is an integral part of the number n/2. These polynomials obey the difference 
equation 

/ d 2 d \ r , . , 

They satisfy the relation 

1 /'CO 

/ jP* e -P 2 / 2 H m (p)dp = r m e- x2 ' 2 H m {x) 

V 27T J_oo 

(see, for example, Subsection 12.2 .4 in [36]), which can be written in the form 

/oo 
e 27Tipx e- np2 H m (V2^p)dp = r m e~™ 2 H m {V2^x). (13.4) 
-00 

Using the Hermite polynomials we create polynomials of many variables 

H m (x) = H mi>m2> ,„ imn (xi,X2,...,x n ) := H mi (x 1 )H m2 (x 2 ) ■ ■ ■ H mn (x n ). (13.5) 
The functions 

e Hx|2/2 #m(x), m, = 0,1,2,..., i = l,2,...,n, (13.6) 
form an orthogonal basis of the Hilbert space L 2 (]R n ) with the scalar product 

</i,/2> := / /i(x)Kx)dx, 
where fix = dx\ dx2 ■ ■ ■ dx n . 
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The polynomials H m (x) satisfy the differential equation 

( A-2^x i A + 2|m|W m (x) = 0, (13.7) 



dxi 
i=i 



where A is the Laplace operator A = jrr~ and | m | = m i + m 2 + • • • + m n - 

ox rn 

We make symmetrization and antisymmetrization of the functions 
W m (x) := e-^H m (V2^x) 
(obtained from (|13.6j) by replacing x by \^2ttx) by means of orbit functions of A n —\: 

I A (x)e- 7r l x l 2 if m (^x) = rl m l e -l A l 2 ^ m (^A), (13.8) 

[ ^ A (x)e-l x l 2 i/ m (^x)=rl m l e -l A l 2 i/r i (^A), (13.9) 

where </>a( x ) is a symmetric orbit function of A n -i, given by formula (6.2) in [I], </?a( x ) is an 
antisymmetric orbit function of A n -i, and A = (Ai, A2, • • • , A re ). 

The polynomials -ffm™ and H^ tl are symmetric and antisymmetric, respectively, with respect 
to the Weyl group W = S n of A n _i: 

HZ m {w\) = HZ m (X), H^(wX) = (det w)H^(X), w e S n . 

For this reason, i?m m (A) are uniquely determined by their values of A = (Ai, A2, . . . , A n ) such 
that Ai > A 2 > • • • > A n and H^^X) by their values of A such that Ai > A 2 > • • • > X n . (Note 
that H^(X) = if Ai = X i+1 for some i = 1, 2, . . . ,n - 1.) 
The polynomials -ffm™ are of the form 

#nT(A) = £ H wm (X), (13.10) 

w<=S„ 

where the polynomials H wm (X) are of the form fj 13.5|) . The polynomials H^ 1 are of the form 
#m nti (A) = (fetw)H wm (X), (13.11) 

that is, 

H™ ti (X)=det(H mi (X j ))l j=1 . 

Moreover, H^ U (X) = if rrii = m^+i for some i = 1,2, ... ,n — 1. For this reason, we may 
consider the polynomials -£fm m (A) for n-tuples m such that mi > 777-2 > • • • > m„ and the 
polynomials JT^J* 1 (A) for n-tuples m such that mi > 7772 > • • • > m n . 

Let us apply symmetric orbit function transform (8.10) of [T] to the symmetric function 
(|13.10p . Taking into account formula (|13.8p we obtain 



d (e^l x l 2 ^ m (v / 2^x)) := [ A (x) e -l x l 2 iY^ m (v / 2^x)dx 

= r^e-^H^iV^x), 



where \S n \ is an order of the permutation group S n , that is, functions (|13.10|) are eigenfunctions 
of the symmetric orbit function transform Since the functions (|13.10p for mj = 0, 1, 2, ... , 
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i = 1,2, ... ,n, mi > m 2 > • • • > m n , form an orthogonal basis of the Hilbert space Lg ym (IR n ) 
of functions from L 2 (M n ) symmetric with respect to W , then they constitute a complete set of 
eigenfunctions of this transform. Thus, this transform has only four eigenvalues i, — i, 1, —1 in 
Lg ym (R n ). This means that, as in the case of the usual Fourier transform, we have 

t = L 

Now we apply antisymmetric orbit function transform (|10. 12j) to the antisymmetric func- 
tion ([13. lip . Taking into account formula ([13.9P we obtain 

= rl m le""l A l 2 ^ nti (v^A), mi > m 2 > • • • > m n > 0, 

that is, functions fj 1 3 . 1 1|) are eigenfunctions of the symmetric orbit function transform 5- Since 
the functions (I13.11|) for mi = 0,1,2,...; i = 1,2, ... ,n, mi > m<i > • • • > m n > 0, form an 
orthogonal basis of the Hilbert space L^ nti (R n ) of functions from L 2 (W l ) antisymmetric with 
respect to W, then they constitute a complete set of eigenfunctions of this transform. Thus, this 
transform has only four eigenvalues i, — i, 1, —1. This means that, as in the case of the usual 
Fourier transform, we have 

S 4 = l. 

13.3 Symmetric and antisymmetric sets of polynomials 

In the previous subsection we constructed symmetric and antisymmetric sets of functions con- 
nected with Hermite polynomials. Similarly other sets of orthogonal polynomials can be con- 
structed (see [37], [38] and [39]). 

Let p m (x), m = 0, 1, 2, . . . , be the set of orthogonal polynomials in one variable such that 

Pm(x)p m '(x)da(x) = S mm >, 

where da(x) is some orthogonality measure, which may be finite or discrete. 
We create a set of symmetric polynomials of n variables as follows: 

Pm m ( x ) = Pm wm {xi)Pm w(2) (x 2 )---Pm Mn) (Xn), (13.12) 

w£Sn/S m 

mi = 0, 1,2, ... , i = 1,2, ... ,n, 

where m = (mi, 77t2, . . . , m n ), mi > m<i >_•••> m n > 0, x = (xi, x-i, . . . , x n ), and w(l),w(2), 
. . . , w(n) is a set of numbers 1,2, ... ,n transformed by the permutation w G S n /S m , where S m 
is the subgroup of S n consisting of elements leaving m invariant. 
We also create the set of polynomials 

Pm tl (x) = ( detw )Pm wW (xi)Pm w(2) (x2)---Pm w(n) (Xn) = det (p m (Xj))ij=l » (13.13) 

wGS„ 

mi = 0, 1,2, ... , i = 1,2, ... ,7i, 

where notations are the same as in (|13.12|) and the condition mi > m-i > ■ ■ ■ > m n > is 
satisfied. 
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It is easy to check that the polynomials Pm m (x) are symmetric with respect to transformations 
of S n : 

p s m m (u;x)=^ m (x), w€S n . 

Similarly, the polynomials (x) are antisymmetric with respect to transformations of S n : 

jCW) = (det (x), w E S n . 

Thus, we may consider the polynomials (|13. 12j) and (|13.13p on the closure of the fundamental 
domain of the transformation group W(A n -i) = S n . This closure (we denote it by D + ) coincides 
with the set of points x = (xi, X2, ■ ■ ■ , x n ) for which 

x\ > X2 > • • • > x n > 0. 

The polynomials (|13.13|) vanish if for some i, i = 1, 2, . . . , n — 1, we have xi — x%j^\. 

The set of polynomials (|13. 121) . as well as the set of the polynomials (|13.13p . is orthogonal 
with respect to the product measure dcr(x) = da(x\) da(x2) ■ ■ ■ da(x n ). Indeed, we have 

/_^ m (x)^f(xT^(x) = ^£^<W = n^<W, 

JD + Pn| Pm| 

f_p: ti (x)tfw^(x) = <w, 

J D+ 

where O(m) is the 5 n -orbit of the point m. 

Note that each polynomial Pm ( x ) vanishes at for any admitted i and j. This means 

that p^ u (x) can be divided by Xi — Xj. Therefore, the functions 

n fa - 

l<i<j<n 

are polynomials in Xi, i = 1,2, ... , n. These polynomials are also orthogonal and the orthogo- 
nality relation is of the form 

l_ P m flti (x)P^ ti (x)H(x)^(x) = cW, 
J D+ 

where H(x) = n (xi — Xj) 2 . 

l<i<j<n 
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